Turning off over commit in the Linux kernel

Linux Malloc's never fails (or at least not until you request more than 150% of the available RAM). This is because the Kernel is optimistic - even if you ask for a lot of memory, maybe you won't use it. Seems like a bad bet, but as long as it's true, everything will run fine. As soon as it's false, the kernel starts killing processes randomly. Yuck.

This is because the swap partition does not grow. If you ask for more RAM than your actual ram PLUS your swap partition, Linux will happily "give" it to you, but has no recourse in the situation that you actually touch all those pages.

In the case that you want Malloc to fail if there's not enough available RAM (such as in embedded code), what to do?

The solution is to modify the kernel, or use syscntrl at run-time. I prefer the permanent solution, so here's how to do it.

In mm/mmap.c, about one page down are two variables which control the "over commit" strategy. Starting around line 80:

int sysctl_overcommit_memory = OVERCOMMIT_NEVER;

/* default = OVERCOMMIT_GUESS; heuristic overcommit */

int sysctl_overcommit_ratio = 0;

/* default is 50% */

Postscript: doing this often results in mallocs failing even when there is clearly lots of RAM remaining. It seems that Linux makes the general assumption that over commit is on, and kind of breaks if it's turned off. A better strategy may be to allow the default (default = OVERCOMMIT_GUESS; heuristic overcommit), and modify the code to be slightly more conservative. I find that heuristic overcommit tends to guess that there is about 2% more RAM available than really is. This would probably vary between machines/kernels/apps running a bit, though I haven't tested it.

Growing Kumquat's in a pot

Kumquats are delicious mini-citrus fruit with a sweet outer rind and a sour flesh. When eaten whole they are a wonderfully sweet, sour, and pungent experience, all in one bite-sized morsel. 

Even better, Kumquats can be grown in a (large) pot, something really not possible with most other types of sweet citrus. 

You can buy the plant from Lowes in San Diego for about $30. I decided to try growing one from seeds. Typically Kumquats are grafted onto hardier rootstock, which is not so easy for a home grower.  But you can also grow them from seed. These general tips on growing indoor fruit trees are also helpful. It's a slow process, but you can also try rooting a cutting.

If your Kumquat never flowers, you may need more humidity. 

Asparagus pasta

Prep time: 10 min with Cuisinart to chop asparagus, otherwise ~20 min. Total time: about 30 min. Serves 2 people, or more if you go heavy on the pasta. 

1lb asparagus	Chop into thin circles, about 1/8th of an inch or thinner.  A Cuisinart makes this a snap!
3 T olive oil (or Butter)	Sauté in pan with oil until asparagus is tender. Butter tastes better than olive oil. Start the pasta
2 cloves crushed garlic 2 T lemon juice 1 T Worcestershire sauce 0.5t pepper 1 t salt angle hair pasta (cooked volume of pasta should roughly equal the volume of the asparagus)	Once all ingredients are mixed in (including pasta) it's ready to serve.  Worcestershire sauce is important to the taste, but soy sauce could be subbed if needed.  salt to taste - 1t is the bare minimum amount  I like to go easy on the pasta - you could easily add more than I suggest here.

You can make it more filling by adding in 1lb of ground turkey. 

(T = table spoon, t = teaspoon)

(c) 2009

Beef stu with beans

This stew is heavy on the veggies, and has zero potatoes (beans fill the starch role).  Yummy!

Prep time: about 1 hour. Total time: about 3 hours, depending on how tender you like the meat. Remember to use a big pot; this makes 2-3 gallons of stew. 

1.5c pinto beans 1.5c red kidney beans	Soak overnight & then simmer for ~ 1 hour
olive oil for pot 2lbs London broil steak 2 medium onions 2-3 garlic cloves	Cut beef into small pieces (< .5in on a side) Chop onions and mince garlic Brown beef in a large pot (cook until water boils off), then add onion and garlic and cook till onion is transparent.
0.25c flour	Add flour and cook until light brown.
beef bouillon (~8000mg sodium) 1t Worcestershire sauce 0.5t ground pepper 0.5c water	Washington's rich brown seasoning and broth is the best, but Wyler's beef bullion is almost as good. I put in enough to get 8g of sodium, but that may be too salty for some tastes. Add all, and mix until bullion is fully dissolved.
7 c water cooked beans	I like to use the water from the beans, which adds extra flavor.  Cook until meat is tender, perhaps 1 hour.
7c carrots  3c celery 1lb frozen peas	Remove peas from freezer to start defrosting them.  7c of carrots is roughly a 2lb bag of carrots.  Chop veggies smaller if you want them to cook faster. If you like crisp veggies, add carrots an hour before you want to eat, celery ~30 minutes before, and peas ~15.

I like to cook the beans a long time so that they split open a bit and thicken up the stew nicely.  If you don't like beans, try it with 8c of potatoes. Finding the right bouillon is key to this recipe. Look for something with MSG and onion powder. Plain old beef flavoring is a bit too dull.

Best OS for low-end laptops: take 3

Just a quick update: less than a month after my two posts on the topic of the best OS for low end machines, somebody posted an Ask Slashdot on the same question.  A quick glance thru the comments suggests that Linux is the most popular option, which isn't surprising given the audience. 

The real value of the Ask Slashdot article, however, is that people are suggesting actual Distros.  So far xubuntu doesn't get a lot of hits, suggesting that perhaps I didn't select the very best option for my own head-to-head tests.  If you do decide to install Linux on your low end laptop, you should check it out:

Ask Slashdot: Best OS For Netbooks and Underpowered Tablets?

Converting Windows filenames to Unicode

Let's say you used FindFirstFile, etc to get a list of file names. What character set will those files be in? Evidence suggests Windows code page 1252.  Now, let's say you want to convert those 8-byte chars into Unicode. One easy way that almost works is: 

wchar_t tmp[256]; // potential buffer overflow exploit here

swprintf(tmp, L"%S", name); // %S (note caps) converts narrow (8 byte) to unicode 

This will fail, sooner or later, because this conversion only seems to work for ASCII defined characters (ie char values less than 127).  WinLatin (1252) has plenty of characters which are not ASCII. You might think you'll never encounter them, but one way that they can creep in is if you use MSWord's smartquotes, and then paste text from Word into your filename.  Probably filenames written by people outside of the US also tend to have these characters.

I don't know what would happen with swprintf if you had some other code page, but I'm guessing it also would be bad.

Here's the proper way to do it. 

wchar_t tmp[1024];

MultiByteToWideChar(CP_ACP,MB_PRECOMPOSED,name,-1,&tmp[0],1023);

Not only does this work for 'fancy' characters, but it also prevents the buffer overflow bug. 

Tribal Wars free browser game review

Tribal wars is a free, real time strategy game that you play from your browser. It feels a bit like single-race WarCraft 2, adapted to play in your browser and running about 100 times slower. Tribal wars (hereafter TW) is completely free, but has a premium mode which makes the game interface easier to use once you become a more powerful player. The game is quite popular, with over 500k players, spread across 15 different servers, each with slightly different rules.

The mechanics of the game are based around villages on a map. You start out with one village, which can produce troops, generate and store resources, etc. You can upgrade the village in lots of ways, and there is also a simple tech tree for your troops, at least on some servers. While there is some strategy in what upgrades you pursue, esp. at first, in the end you'll want most of them. One key aspect of the game is that everything takes time, from upgrading, to building troops, to sending the troops to other villages . Early upgrades take 30 minutes. Later, an upgrade can take 12 or more hours. Nearby villages can be reached by your troops in an hour, but soon you'll be attacking villages that are 5 hours away, and eventually you'll send your troops on attacks that take more than a day to reach their destination. This means that you'll never have a marathon 5-hour session of TW. At most, you'll log in for an hour at a time to upgrade you village, build up some troops, and launch a new attack. But, you'll need to log back in again later that day to keep things on track. Generally, 3 logins a day is sufficient. You can start to see how this game can be time consuming, and not so friendly to going on vacation. Luckily, there's a way to have another player in the game "account-sit", which means they can temporarily play as you, but don't need your password to login.

The other major aspect of the game is the tribes. While you can play alone, you won't survive long. You'll need to join a tribe, which I suggest you do early on, as it's a good place to ask for advice. Your tribe is made up of other players in the game, which you can email using the in-game mail system, or chat with using the in-game forum. The forum gets a lot of use, for planning attacks and co-ordinating on-going attacks. There's the potential for a lot of diplomacy in this game, as the only way you grow, more or less, is by taking over other people's villages. The art of it all is selecting villages you can take over with minimal risk to yourself and your tribe, and likewise, preventing other tribes from attacking you. Generally lots of alliances, and other loose agreements are made between tribes, and this aspect is as important as growing your army. While the game is most certainly PvP, it's important to note that there are protections that prevent new players from being wiped out, and really strong players from killing really weak players. The art of growing your empire in this game is selecting opponents who are weaker than you, but not too weak, and have a puny tribe.

Graphically, the game is good enough - not spartan, but definitely not flashy. If it weren't for the real-time aspects of the game, I would say it's a lot like playing a board-game with 30 other people (since the world is so large, you don't tend to interact with players that are more than 2 days away from you, in terms of time it would take to send an army).

Finally, I would like to stress that the focus of the game changes a bit as you play it, and your empire grows. The changing focus helps keep the game fresh, at least for a while.

I played the game for about 6 months, during which it went through about 3 distinct phases. At first, during the first 2 months, I spent all my time growing my one village, learning about the upgrades, and farming (sending attacks to NPC villages which do not fight back, but which produce resources that you can steal). New villages are started at the edge of the map, which slowly grows as more players are added. Thus, at first you are surrounded by players who are all the same strength as you, and all very weak.

Sometime in my second month I was able to take over my first village from another player, but the dynamic didn't change much. Once I had about 4 villages, however, I was able to start using them together to make it much easier to take more villages. This does change the strategy a bit. There were still lots of easy targets, because most of the players around me still didn't have strong tribes to defend them.

After another 2 months or so, however, every village/player was a member of a tribe, some much larger than mine, and some a bit smaller. At this point, the diplomacy aspect of the game became really key, because you need the other members of the tribe to help you gang up on the tribe you want to attack next. You can no-longer go it alone. I think this was the most fun part of the game. Around this time, it became more or less impossible to play for free, because you really need the premium features that make it easier to manage all those villages. Premium costs about $5 a month.

Note that I say was. After about 6 months I found the game was starting to lose its appeal. It got a bit tiresome to manage all 30 villages. I was spending about 2 hours a day on the game, and a lot of that time felt like work: logging in to build troops, upgrade my new villages, etc. Not so much of the time was spent on the strategy of deciding who to attack, and which alliances to strengthen.

I decided to quit playing at that point, though I don't regret the time I put into the game up till then.