Thursday, March 30, 2006
Now it's the time...
...of game programming!
So far, you've been told about what we do here at Unkasoft: our tools, methodology and the problems we're solving every day. However, what would a game studio be, if we didn't do games?
We've been working hard for the last five months, in order to complete our first game: 'The Fourth Seal'. It's been a lot of work because, as we were making the game, we had to deal with Battlewizard's bugs, as well as the game's ones: we were testing our system for the very first time. We were developing a great game, but also improving BW's engine and its implementations for a wide range of devices. You could say 'blood, sweat and tears'.
But it hasn't been all that bad; we've gained a lot of experience that has given us some clues about what can or can't be done and how to face some situations. As a result, now we've got our own game template, which is being used in the development of our second game, 'Isometric Roller'. This template includes a basic application skeleton and some of the data structures and algorithms created for 'The Fourth Seal'. It's making the process far easier, so we're expecting to be able to push BW to its limits from now on, offering you lots of great and, the most important, FUN games.
So far, you've been told about what we do here at Unkasoft: our tools, methodology and the problems we're solving every day. However, what would a game studio be, if we didn't do games?
We've been working hard for the last five months, in order to complete our first game: 'The Fourth Seal'. It's been a lot of work because, as we were making the game, we had to deal with Battlewizard's bugs, as well as the game's ones: we were testing our system for the very first time. We were developing a great game, but also improving BW's engine and its implementations for a wide range of devices. You could say 'blood, sweat and tears'.
But it hasn't been all that bad; we've gained a lot of experience that has given us some clues about what can or can't be done and how to face some situations. As a result, now we've got our own game template, which is being used in the development of our second game, 'Isometric Roller'. This template includes a basic application skeleton and some of the data structures and algorithms created for 'The Fourth Seal'. It's making the process far easier, so we're expecting to be able to push BW to its limits from now on, offering you lots of great and, the most important, FUN games.
The building process
Few years ago, I read an article of Joel on Software about daily builds. In that time, it was Greek to me, when I thought building a program was only picking "Project - Build" option inside my IDE.
However, after few years, I realize that "compiling a programing" has nothing to do with "building a product", although both should be done with only one step.
Compiling a program means only convert source code into binary executable code, and that's a repetitive and mechanical work (although quite complex) solved by compiler. That's why it was invented isn't it?
Even though, building a product involve much more. For example, in our situation, in order to build The Fourth Seal, we have following tasks:
In other side we have Battlewizard building, with its own steps, quite different, mainly because Battlewizard has a desktop application with its installer, executable file, etc.
Perhaps we forgot something, but it's just a sample and we get used to the idea of it.
All these steps we described form product building process, however, we need something more: it should be automatic. Yes, the most important thing about it is building process must be:
Another benefit: building completely and periodically our product, we can detect wrong code very early. Let's assume one of our libraries hasn't been modified for months, and when we're going to compile it and run its unit tests, it fails. If we would compile and test that library every day or week (together with the rest of the product), that error would be detected earlier, a few hours or days after someone introduced it, and it would be easier and faster to fix it.
And finally, making automatic building, we'll allow everybody, even new team members, will be able to generate all product components. Building process will be responsible to get file from somewhere, doing necesary copies, modify whatever libraries and so on, but user only makes a simple step.
Well, tomorrow we'll talk about famouse "continuous integration" recommended for agile methodologies like eXtreme Programming.
Meanwhile, you go further in this subject reading these recommended issues:
However, after few years, I realize that "compiling a programing" has nothing to do with "building a product", although both should be done with only one step.
Compiling a program means only convert source code into binary executable code, and that's a repetitive and mechanical work (although quite complex) solved by compiler. That's why it was invented isn't it?
Even though, building a product involve much more. For example, in our situation, in order to build The Fourth Seal, we have following tasks:
- Compile the Battlewizard abstract engine.
- Compile specific implementation for desired handset: MIDP 2.0, nokia and so on. (By the way: I owe you one explanation about Battlewizard and its architecture :)
- Increment current version and create manifest file
- Pack everything inside a JAR file.
- Compile game source code, using specific resources (images, sound effects, text strings, etc) for desired device and language.
- Launch unit tests for all our code base (using JUnit and J2MEUnit)
- Obfuscate everything (using ProGuard or RetroGuard, depending on current situation)
- Optimize and compress resulting bytecode (with specified libraries)
- Create JAD file with JAR file size, MIDlet name, and so on.
In other side we have Battlewizard building, with its own steps, quite different, mainly because Battlewizard has a desktop application with its installer, executable file, etc.
Perhaps we forgot something, but it's just a sample and we get used to the idea of it.
All these steps we described form product building process, however, we need something more: it should be automatic. Yes, the most important thing about it is building process must be:
- automatic, without human intervention, or at least with minimum intervention (you can ask for data to the user, like asking if version should be demo or release, for example)
- as simple and easy as a single click (ok, we'll admit running a console command or even a double click :)
- complete (it's invalid recompile only certain libraries or use incremental compilation)
- and whenever possible: daily, launched every nigh (don't you know famous nightly builds?) or at least run in a periodical way.
Another benefit: building completely and periodically our product, we can detect wrong code very early. Let's assume one of our libraries hasn't been modified for months, and when we're going to compile it and run its unit tests, it fails. If we would compile and test that library every day or week (together with the rest of the product), that error would be detected earlier, a few hours or days after someone introduced it, and it would be easier and faster to fix it.
And finally, making automatic building, we'll allow everybody, even new team members, will be able to generate all product components. Building process will be responsible to get file from somewhere, doing necesary copies, modify whatever libraries and so on, but user only makes a simple step.
Well, tomorrow we'll talk about famouse "continuous integration" recommended for agile methodologies like eXtreme Programming.
Meanwhile, you go further in this subject reading these recommended issues:
- Daily Builds Are Your Friend from Joel on Software, where I began to discover the building world.
- Daily Build and Smoke Test from Steve McConnell. He talks about the approach used in Microsoft for daily builds and QA process.
Friday, March 24, 2006
Size doesn't matter
This week we've been in two conferences: Sun's Tech Days on Tuesday and the presentation of new configuration management product yesterday.
Meanwhile speakers show us all their products, I was realizing how difficult and different is game programming. Relational databases, O/R mapping frameworks, service based architectures, AJAX, OpenGL... and here we're fighting against bit shifting and juggling with three dimensions arrays which explodes our heap.
As Joel sais, all programmers write code, but each programmer in his/her own different world.
I'm going to tell you a real story, which Jaime has mentioned yesterday: our games are quite
big, specially taking into account memory capabilities of few handsets we support. The Fourth Seal has 12 levels and 9 different enemies, so it put our game at the level of some PC games, but running inside a small phone with 500 KB heap. This force us to squeeze phone memory and, as Jaime said, there're many things in heap when it mustn't be here.
If you write desktop or server applications, probably you don't realize if you're wasting your memory. It will be just enough detecting and fixing memory leaks, and ensuring memory requirements don't raise too much.
But in our world it's really important.
One thing we has in memory and hindered us, was executable instructions (bytecodes) of some methods. We have a lot of preprocessing code, which is responsible of create offscreen images, read data from disk, sort lists, decompress data, etc. and all that code only is executed once, duing game startup, so we wanted to took that load off our shoulders when it wasn't needed any more.
But... how? Java (and of course J2ME) do not offer any way to unload memory of one specified method... well, there aren't any direct way, but it isn't all lost.
Our solution goes tru joining all that "use and throw out" code in one single class. When first class instance is created, all bytecode will be loaded in memory and when last instance is released and garbage collector called, all those bytecodes will be freed from heap. Simple and easy.
Getting memory graphs (like you can see up there), you can see two sudden memory drops. It is just the moment when last "loader class" instance is freed and garbage collector invoked.
As it's used to say: size doesn't matter, except if you're programming phone games!!
Meanwhile speakers show us all their products, I was realizing how difficult and different is game programming. Relational databases, O/R mapping frameworks, service based architectures, AJAX, OpenGL... and here we're fighting against bit shifting and juggling with three dimensions arrays which explodes our heap.
As Joel sais, all programmers write code, but each programmer in his/her own different world.
I'm going to tell you a real story, which Jaime has mentioned yesterday: our games are quite
big, specially taking into account memory capabilities of few handsets we support. The Fourth Seal has 12 levels and 9 different enemies, so it put our game at the level of some PC games, but running inside a small phone with 500 KB heap. This force us to squeeze phone memory and, as Jaime said, there're many things in heap when it mustn't be here.If you write desktop or server applications, probably you don't realize if you're wasting your memory. It will be just enough detecting and fixing memory leaks, and ensuring memory requirements don't raise too much.
But in our world it's really important.
One thing we has in memory and hindered us, was executable instructions (bytecodes) of some methods. We have a lot of preprocessing code, which is responsible of create offscreen images, read data from disk, sort lists, decompress data, etc. and all that code only is executed once, duing game startup, so we wanted to took that load off our shoulders when it wasn't needed any more.
But... how? Java (and of course J2ME) do not offer any way to unload memory of one specified method... well, there aren't any direct way, but it isn't all lost.
Our solution goes tru joining all that "use and throw out" code in one single class. When first class instance is created, all bytecode will be loaded in memory and when last instance is released and garbage collector called, all those bytecodes will be freed from heap. Simple and easy.
Getting memory graphs (like you can see up there), you can see two sudden memory drops. It is just the moment when last "loader class" instance is freed and garbage collector invoked.
As it's used to say: size doesn't matter, except if you're programming phone games!!
Thursday, March 23, 2006
Vicious Circle
Once again the problem from HEAP memory and the JAR file (J2ME) size with the resources. One of the biggest problems we have every day with the most limited handsets (mobile phones), is that compressing our files to the maximum is not enough. This is because although we can resize our JAR files for solving our JAR limit we found that our resource will be decompressed in our HEAP space and we reach the top of our HEAP too early.
The real problem with the memory from mobile phones is that we never have the expected amount. You can read that you have n KB available from HEAP, but we never think about decompressing our images (PNG...) as bitmaps in memory, that our jar file need to be decompressed, that our java classes need to be loaded into memory and then when we run our application we found that we are using more memory than expected from our game logic.
At the end, the solution is a question of balance. If you reduce the size of our JAR package compressing to the maximum and you don’t think about content, when you decompress you will find HEAP limitations. The best way is to think about it from the beginning, with all resources and the order of creating and destroying them in memory.
I am going to enumerate some rules (I think important) that we use when we evaluate the HEAP that we are using with our resources and its working for us with good results:
1 – Never think in the top of the HEAP memory specified by the manufacturer. You will have always less memory in the handset than the expected. This can be because of firmware bugs or by internal use from manufacturer implementation.
2 – Your images prepared for painting will use the amount of memory from the formula:
Image (KB) = (WIDTH (pixels) x HEIGHT (pixels) x COLOURS (bytes)) / 1024
For an image of 64 x 64 in a screen of 65.535 colours (16 bits = 2 bytes) we will have
(64 x 64 x 2) / 1024 = 8 KB from HEAP.
Try to optimize the size of the image for your HEAP and the number of colours in the image for your JAR file.
3 – The code of your classes could have to live in the HEAP. Remember that using a class need of its bytecode in somewhere. Usually the place of storing the bytecode of the class in use will be the HEAP. Think that decompressed classes will use the HEAP when you use them. A good solution could be to have a specific class for one use methods, like loading and preparing. When you finished using those methods, you can set your objects to null for some mobile phones to remove the bytecode from the HEAP.
4 – If you think that your JAR file is not going to be decompressed in the HEAP, you are totally mistaken. Some mobile phones will decompress your HAR file in your HEAP and if you have compressed to the maximum, you will find not too much available HEAP. Try to balance the content from your JAR file with the necessary HEAP from your game. No balancing is equal to enter in a vicious circle between JAR and HEAP. Other solution is to implement a small decompressor for some of your resources (no classes, no images and no sounds), allowing you to maintain them compressed into memory. Balancing is essential.
5 – Most of the resources and classes have headers that need HEAP and JAR space. The lower number of classes and resources files, the lower space you will use. Beware of creating all of your game logic in one unique class or inserting all of your graphics in one image, because using a too big resource or class could reach the top of the HEAP. Again, try to balance between the number of classes and the number of resources files. This is more complex because we need to analyze the game logic for making the right division, but will make easy working in small mobile phones.
6 – Adapt the layout of your graphics inside of your image resources. It doesn’t take up into memory using a more vertical image that a horizontal or a quad image. From a long time ago (old 8 bits games) we know that the layout of the graphics in the images cab use more or less HEAP memory. Sometimes setting your sprites frames into vertical layout (one column) could improve you memory usage. Experiment until you find the best layout and combine it with your number of resource images balancing.
7 – Try to use image optimizing tools like PNGOUT, JAR file optimizing tools like JARG (Java Archive Grinder) or KZIP. With these tools you will improve your JAR file size, but remember balancing with your HEAP space. Using obfuscator tools is mandatory for improving JAR files size and bytecode HEAP usage. Some obfuscators optimize the size better than others but some mobile phones don’t like big classes with some kinds of obfuscation. Try to use the more adequate obfuscator in each case.
8 – Preloading of images and sounds can accelerate the execution of your games but can also limit the game logic to store in your HEAP. Balance your resources cached into memory with your game logic. Although not all of your resources will be cached, you can make some effects during the game play for the user not to fell the loading and preparing times. Load and unload resources from HEAP memory and try to use some tricks during the game for the user. Loading from JAR files is slow and having images (bitmap) into memory prepared for painting is expensive for the HEAP. A good idea is having some image files (PNG...) into memory arrays like if you have a cache, and when you need them, prepare them as bitmaps for painting. This is faster than getting them from jar file and its size in the HEAP memory is the same as the image file compressed in their original format. Same can be done with some audio resources.
9 – Your mobile phone has a persistent memory and in the case of J2ME we can save data using the Record Store. In some cases if your resources and logic don’t fit in your JAR file limit, you can try to add the game logic to the JAR file and initializing the game, the game can download some of the resources storing them into the Record Store. We are using mobile phones that can connect with Internet and we must use these capabilities for improving our games. Gradually the players will assimilate that mobile applications need of connecting to the Internet for updating and others. Creating some Record Store can be a slow action but reading and writing records in a Record Store is enough fast. Order your contents in a way that you can load them without strange feelings, like stopping the action, to the player. You could create cool games in limited mobile phones. Remember that there are some limits for Record Store from a MIDlet.
10 – Balancing and designing. Everything is a question of balancing and depends on how we design our game. If we are able to design basing in the balance between JAR and HEAP, we will obtain to insert a big quantity of contents and we will create levels and screens very complete for the player. When you are making a game for a mobile phone, you must think always in the balance for avoiding headaches. This rule is valid for all platforms because at the end is always a problem of capabilities. The more content, the more balancing work we need.
The real problem with the memory from mobile phones is that we never have the expected amount. You can read that you have n KB available from HEAP, but we never think about decompressing our images (PNG...) as bitmaps in memory, that our jar file need to be decompressed, that our java classes need to be loaded into memory and then when we run our application we found that we are using more memory than expected from our game logic.
At the end, the solution is a question of balance. If you reduce the size of our JAR package compressing to the maximum and you don’t think about content, when you decompress you will find HEAP limitations. The best way is to think about it from the beginning, with all resources and the order of creating and destroying them in memory.
I am going to enumerate some rules (I think important) that we use when we evaluate the HEAP that we are using with our resources and its working for us with good results:
1 – Never think in the top of the HEAP memory specified by the manufacturer. You will have always less memory in the handset than the expected. This can be because of firmware bugs or by internal use from manufacturer implementation.
2 – Your images prepared for painting will use the amount of memory from the formula:
Image (KB) = (WIDTH (pixels) x HEIGHT (pixels) x COLOURS (bytes)) / 1024
For an image of 64 x 64 in a screen of 65.535 colours (16 bits = 2 bytes) we will have
(64 x 64 x 2) / 1024 = 8 KB from HEAP.
Try to optimize the size of the image for your HEAP and the number of colours in the image for your JAR file.
3 – The code of your classes could have to live in the HEAP. Remember that using a class need of its bytecode in somewhere. Usually the place of storing the bytecode of the class in use will be the HEAP. Think that decompressed classes will use the HEAP when you use them. A good solution could be to have a specific class for one use methods, like loading and preparing. When you finished using those methods, you can set your objects to null for some mobile phones to remove the bytecode from the HEAP.
4 – If you think that your JAR file is not going to be decompressed in the HEAP, you are totally mistaken. Some mobile phones will decompress your HAR file in your HEAP and if you have compressed to the maximum, you will find not too much available HEAP. Try to balance the content from your JAR file with the necessary HEAP from your game. No balancing is equal to enter in a vicious circle between JAR and HEAP. Other solution is to implement a small decompressor for some of your resources (no classes, no images and no sounds), allowing you to maintain them compressed into memory. Balancing is essential.
5 – Most of the resources and classes have headers that need HEAP and JAR space. The lower number of classes and resources files, the lower space you will use. Beware of creating all of your game logic in one unique class or inserting all of your graphics in one image, because using a too big resource or class could reach the top of the HEAP. Again, try to balance between the number of classes and the number of resources files. This is more complex because we need to analyze the game logic for making the right division, but will make easy working in small mobile phones.
6 – Adapt the layout of your graphics inside of your image resources. It doesn’t take up into memory using a more vertical image that a horizontal or a quad image. From a long time ago (old 8 bits games) we know that the layout of the graphics in the images cab use more or less HEAP memory. Sometimes setting your sprites frames into vertical layout (one column) could improve you memory usage. Experiment until you find the best layout and combine it with your number of resource images balancing.
7 – Try to use image optimizing tools like PNGOUT, JAR file optimizing tools like JARG (Java Archive Grinder) or KZIP. With these tools you will improve your JAR file size, but remember balancing with your HEAP space. Using obfuscator tools is mandatory for improving JAR files size and bytecode HEAP usage. Some obfuscators optimize the size better than others but some mobile phones don’t like big classes with some kinds of obfuscation. Try to use the more adequate obfuscator in each case.
8 – Preloading of images and sounds can accelerate the execution of your games but can also limit the game logic to store in your HEAP. Balance your resources cached into memory with your game logic. Although not all of your resources will be cached, you can make some effects during the game play for the user not to fell the loading and preparing times. Load and unload resources from HEAP memory and try to use some tricks during the game for the user. Loading from JAR files is slow and having images (bitmap) into memory prepared for painting is expensive for the HEAP. A good idea is having some image files (PNG...) into memory arrays like if you have a cache, and when you need them, prepare them as bitmaps for painting. This is faster than getting them from jar file and its size in the HEAP memory is the same as the image file compressed in their original format. Same can be done with some audio resources.
9 – Your mobile phone has a persistent memory and in the case of J2ME we can save data using the Record Store. In some cases if your resources and logic don’t fit in your JAR file limit, you can try to add the game logic to the JAR file and initializing the game, the game can download some of the resources storing them into the Record Store. We are using mobile phones that can connect with Internet and we must use these capabilities for improving our games. Gradually the players will assimilate that mobile applications need of connecting to the Internet for updating and others. Creating some Record Store can be a slow action but reading and writing records in a Record Store is enough fast. Order your contents in a way that you can load them without strange feelings, like stopping the action, to the player. You could create cool games in limited mobile phones. Remember that there are some limits for Record Store from a MIDlet.
10 – Balancing and designing. Everything is a question of balancing and depends on how we design our game. If we are able to design basing in the balance between JAR and HEAP, we will obtain to insert a big quantity of contents and we will create levels and screens very complete for the player. When you are making a game for a mobile phone, you must think always in the balance for avoiding headaches. This rule is valid for all platforms because at the end is always a problem of capabilities. The more content, the more balancing work we need.
Wednesday, March 15, 2006
How we're ordered
Here at Unkasoft we've divided our development in two quite different sections:
- On the one hand, we've got a tools departament, where we work in our product called Battlewizard. Soon, we'll tell you more about it (you'll get bored :) but in advance I'll tell you it's a complete system to develop mobiles games avoiding the hell of device portability between different vendors, screen sizes, capabilities, etc.
- In the other hand, we've got the games division, where we use Battlewizard and its J2ME API to develop our final games (one of them in test phase, two more in programming and another one in design phase). This allows us to mantain only one single code base for all target devices, and our final games runs in a wide range of phones (this is one of the keys sides that all operators take into account). This isn't usual in mobile development world, even there're dev firms (like GameLoft) that have one codebase for each target device, and each codebase with its own manager. At small studies like ours, that schema isn't feasible and we think that isn't the best approach to fragmentation problem.
That's all about programming.
Then, we've got a QA department, where we test both Battlewizard and our final games for all supported devices.
This is a real muddle! Think about test the same game at 30 different phones (our target is to reach up to 200 supported devices!). You should see Jorge while he's testing two versions of The Fourth Seal with one phone in each hand :)
- On the one hand, we've got a tools departament, where we work in our product called Battlewizard. Soon, we'll tell you more about it (you'll get bored :) but in advance I'll tell you it's a complete system to develop mobiles games avoiding the hell of device portability between different vendors, screen sizes, capabilities, etc.
- In the other hand, we've got the games division, where we use Battlewizard and its J2ME API to develop our final games (one of them in test phase, two more in programming and another one in design phase). This allows us to mantain only one single code base for all target devices, and our final games runs in a wide range of phones (this is one of the keys sides that all operators take into account). This isn't usual in mobile development world, even there're dev firms (like GameLoft) that have one codebase for each target device, and each codebase with its own manager. At small studies like ours, that schema isn't feasible and we think that isn't the best approach to fragmentation problem.
That's all about programming.
Then, we've got a QA department, where we test both Battlewizard and our final games for all supported devices.
This is a real muddle! Think about test the same game at 30 different phones (our target is to reach up to 200 supported devices!). You should see Jorge while he's testing two versions of The Fourth Seal with one phone in each hand :)
Tuesday, March 14, 2006
Unkasoft is something special
Even if this can sound like propaganda brochure, I assure you that working at Unkasoft is something special.
One year ago, my experience in development world says me it was impossible to make good software in Spain, due we got worried about compare us with the others instead of watching our own source code and improve our team progamming abilites (the team, what strange word!).
After working at few companies in the software industry, I realized that current prospect is completly wrong.
During interviews with Jaime, I started to feel something special, both in tech and humane point of view. Questions was really smart and he seemed to worry about their employees. When interview finished, I was thinking about one idea: these people knows a lot about software industry. I feed up with pseudo-interviews in which I ask more than interviewer, and I always came to the conclusion that it was another company in which the only important thing is your experience with this or that language or tool.
Lately, I realize that those "good smells" I felt during the interview are true, and it's possible to work with agile methodologies and following Joel style without waste my time convincing and persuading managers about this is the right way. And, to top it off, working in game development world (always so enigmatic), in an emerging industry like mobiles phones.
In short, working at Unkasoft is something special, and this is the luck of all we work in this small start-up
From here, we'll tell you our day by day, our troubles (a lot of them), failures (I'm sure we'll suffer some of them) and our successes (I hope more than failures :)
We'll tell you about agile methodologies, game programming, mobiles phones, J2ME and also something about our own products.
By the way, I used to be called JM, and I'm one of Unkasoft's programmers.
One year ago, my experience in development world says me it was impossible to make good software in Spain, due we got worried about compare us with the others instead of watching our own source code and improve our team progamming abilites (the team, what strange word!).
After working at few companies in the software industry, I realized that current prospect is completly wrong.
During interviews with Jaime, I started to feel something special, both in tech and humane point of view. Questions was really smart and he seemed to worry about their employees. When interview finished, I was thinking about one idea: these people knows a lot about software industry. I feed up with pseudo-interviews in which I ask more than interviewer, and I always came to the conclusion that it was another company in which the only important thing is your experience with this or that language or tool.
Lately, I realize that those "good smells" I felt during the interview are true, and it's possible to work with agile methodologies and following Joel style without waste my time convincing and persuading managers about this is the right way. And, to top it off, working in game development world (always so enigmatic), in an emerging industry like mobiles phones.
In short, working at Unkasoft is something special, and this is the luck of all we work in this small start-up
From here, we'll tell you our day by day, our troubles (a lot of them), failures (I'm sure we'll suffer some of them) and our successes (I hope more than failures :)
We'll tell you about agile methodologies, game programming, mobiles phones, J2ME and also something about our own products.
By the way, I used to be called JM, and I'm one of Unkasoft's programmers.
