Repost: Fire and Motion

Original Post: http://www.joelonsoftware.com/articles/fog0000000339.html

Joel on Software

Fire And Motion

by Joel Spolsky

Sunday, January 06, 2002

Sometimes I just can't get anything done.

Sure, I come into the office, putter around, check my email every ten seconds, read the web, even do a few brainless tasks like paying the American Express bill. But getting back into the flow of writing code just doesn't happen.

These bouts of unproductiveness usually last for a day or two. But there have been times in my career as a developer when I went for weeks at a time without being able to get anything done. As they say, I'm not in flow. I'm not in the zone. I'm not anywhere.

Everybody has mood swings; for some people they are mild, for others, they can be more pronounced or even dysfunctional. And the unproductive periods do seem to correlate somewhat with gloomier moods.

It makes me think of those researchers who say that basically peoplecan't control what they eat, so any attempt to diet is bound to be short term and they will always yoyo back to their natural weight. Maybe as a software developer I really can't control when I'm productive, and I just have to take the slow times with the fast times and hope that they average out to enough lines of code to make me employable.

     

What drives me crazy is that ever since my first job I've realized that as a developer, I usually average about two or three hours a day of productive coding. When I had a summer internship at Microsoft, a fellow intern told me he was actually only going into work from 12 to 5 every day. Five hours, minus lunch, and his team loved him because he still managed to get a lot more done than average. I've found the same thing to be true. I feel a little bit guilty when I see how hard everybody else seems to be working, and I get about two or three quality hours in a day, and still I've always been one of the most productive members of the team. That's probably why when Peopleware and XP insist on eliminating overtime and working strictly 40 hour weeks, they do so secure in the knowledge that this won't reduce a team's output.

But it's not the days when I "only" get two hours of work done that worry me. It's the days when I can't do anything.

I've thought about this a lot. I tried to remember the time when I got the most work done in my career. It was probably when Microsoft moved me into a beautiful, plush new office with large picture windows overlooking a pretty stone courtyard full of cherry trees in bloom. Everything was clicking. For months I worked nonstop grinding out the detailed specification for Excel Basic -- a monumental ream of paper going into incredible detail covering a gigantic object model and programming environment. I literally never stopped. When I had to go to Boston for MacWorld I took a laptop with me, and documented the Window class sitting on a pleasant terrace at HBS.

Once you get into flow it's not too hard to keep going. Many of my days go like this: (1) get into work (2) check email, read the web, etc. (3) decide that I might as well have lunch before getting to work (4) get back from lunch (5) check email, read the web, etc. (6) finally decide that I've got to get started (7) check email, read the web, etc. (8) decide again that I really have to get started (9) launch the damn editor and (10) write code nonstop until I don't realize that it's already 7:30 pm.

Somewhere between step 8 and step 9 there seems to be a bug, because I can't always make it across that chasm. For me, just getting started is the only hard thing. An object at rest tends to remain at rest. There's something incredible heavy in my brain that is extremely hard to get up to speed, but once it's rolling at full speed, it takes no effort to keep it going. Like a bicycle decked out for a cross-country, self-supported bike trip -- when you first start riding a bike with all that gear, it's hard to believe how much work it takes to get rolling, but once you are rolling, it feels just as easy as riding a bike without any gear.

Maybe this is the key to productivity: just getting started. Maybe when pair programming works it works because when you schedule a pair programming session with your buddy, you force each other to get started.

When I was an Israeli paratrooper a general stopped by to give us a little speech about strategy. In infantry battles, he told us, there is only one strategy: Fire and Motion. You move towards the enemy while firing your weapon. The firing forces him to keep his head down so he can't fire at you. (That's what the soldiers mean when they shout "cover me." It means, "fire at our enemy so he has to duck and can't fire at me while I run across this street, here." It works.)  The motion allows you to conquer territory and get closer to your enemy, where your shots are much more likely to hit their target. If you're not moving, the enemy gets to decide what happens, which is not a good thing. If you're not firing, the enemy will fire at you, pinning you down.

I remembered this for a long time. I noticed how almost every kind of military strategy, from air force dogfights to large scale naval maneuvers, is based on the idea of Fire and Motion. It took me another fifteen years to realize that the principle of Fire and Motion is how you get things done in life. You have to move forward a little bit, every day. It doesn't matter if your code is lame and buggy and nobody wants it. If you are moving forward, writing code and fixing bugs constantly, time is on your side. Watch out when your competition fires at you. Do they just want to force you to keep busy reacting to their volleys, so you can't move forward?

Think of the history of data access strategies to come out of Microsoft. ODBC, RDO, DAO, ADO, OLEDB, now ADO.NET - All New! Are these technological imperatives? The result of an incompetent design group that needs to reinvent data access every goddamn year? (That's probably it, actually.) But the end result is just cover fire. The competition has no choice but to spend all their time porting and keeping up, time that they can't spend writing new features. Look closely at the software landscape. The companies that do well are the ones who rely least on big companies and don't have to spend all their cycles catching up and reimplementing and fixing bugs that crop up only on Windows XP. The companies who stumble are the ones who spend too much time reading tea leaves to figure out the future direction of Microsoft. People get worried about .NET and decide to rewrite their whole architecture for .NET because they think they have to. Microsoft is shooting at you, and it's just cover fire so that they can move forward and you can't, because this is how the game is played, Bubby. Are you going to support Hailstorm? SOAP? RDF? Are you supporting it because your customers need it, or because someone is firing at you and you feel like you have to respond? The sales teams of the big companies understand cover fire. They go into their customers and say, "OK, you don't have to buy from us. Buy from the best vendor. But make sure that you get a product that supports (XML / SOAP / CDE / J2EE) because otherwise you'll be Locked In The Trunk." Then when the little companies try to sell into that account, all they hear is obedient CTOs parrotting "Do you have J2EE?" And they have to waste all their time building in J2EE even if it doesn't really make any sales, and gives them no opportunity to distinguish themselves. It's a checkbox feature -- you do it because you need the checkbox saying you have it, but nobody will use it or needs it. And it's cover fire.

Fire and Motion, for small companies like mine, means two things. You have to have time on your side, and you have to move forward every day. Sooner or later you will win. All I managed to do yesterday is improve the color scheme in FogBUGZ just a little bit. That's OK. It's getting better all the time. Every day our software is better and better and we have more and more customers and that's all that matters. Until we're a company the size of Oracle, we don't have to think about grand strategies. We just have to come in every morning and somehow, launch the editor.

Npm, AngularJS project problem and solution

1. ERROR: EACCES when running npm related command
use whoami  to check the current user, then do
sudo chown -R [username] [targetdir]

2. tar.unpack untar error
upgrade node to the latest version

Good website

1. MKyong
http://www.mkyong.com/

Javascript Optimization

1. Measuring JavaScript Functions' Performance
http://www.sitepoint.com/measuring-javascript-functions-performance/?utm_source=javascriptweekly&utm_medium=email

5 Useful Mac OS X Command Line Tips Everyone Should Know (Repost)

Original Link:
http://osxdaily.com/2008/06/29/5-useful-mac-os-x-command-line-tips-you-should-know/

Like many other power users, I’m addicted to the Mac OS X command line, any reason at all for me to launch the terminal I take as an opportunity to learn more about the powerful backend of our favorite operating system.

Here I have gathered 5 useful commands that will make your life easier when you’re working in the command line interface of OS X, so launch the Terminal and try them out on your Mac! If you have any others that you think should be added to this list, feel free to post them in the comments, and be sure to check out 10 OS X Command line utilities you might not know about for more.

1: Delete an entire line of commands/text

Don’t repeatedly hit the delete key, all you need to do to clear your prompt is hit Control-U and your current prompt will be clean.

2: Create a nested directory structure immediately

If you need to create the directory structure /annoyingly/long/and/outrageous/directory/path/ , instead of typing mkdir annoyingly, cd annoyingly, mk long , etc etc, simply type the following:

mkdir -p /annoyingly/long/and/outrageous/directory/path/

And your nested directory structure will be created in full immediately!

3: Clear the entire Terminal screen

If you have a screen full of nonsense, clearing the Terminal screen is very easy, you can either type:

clear

Or you can just hit the command keystroke Control-Land you’ll have a clean slate to work with.

4: Run a process in the background

If you want to set a process to run in the background, simply put a & after it, the command will execute but you’ll still be in the same shell, allowing you to continue as usual.

For example:

./crazyscript.sh &

Would run that script in the background, and return you right back to your shell.

5: Run the last executed command

Need to re-run the last executed command? ! is the way to go, here are two ways to use it:

First, just typing:

!!

will run whatever the last command that was executed, typing

!l

will run the last command starting with the letter l, and so forth. Very useful, isn’t it?

SCSS

1. Get started with SCSS
http://alistapart.com/article/getting-started-with-sass

What's the best way to avoid garbage collector pauses with big heaps in Java (Question excerpt from Quora)

Answered by Li Pi

https://www.quora.com/Whats-the-best-way-to-avoid-garbage-collector-pauses-with-big-heaps-in-Java

If the GC (Concurrent-Mark-Sweep) (see How does garbage collection work in the JVM?) is operating as expected, then the stop the world pauses should not be significant. CMS is designed to have as few stop the world pauses as possible. The Parallel New Collector will stop the world, but as long as your young generation is reasonably sized, you should be fine.

To recap, CMS operates by first stopping the world, then initially markingthe root nodes, then it concurrently proceeds to trace through the rest of the objects. Memory is freed at the end of this process, but objects are never moved around, making fragmentation problematic.

However, there exist a few failure modes where CMS will be forced the stop the world for a significant amount of time:

1. Concurrent Mode Failure - This occurs when the tenured generation fills up before CMS has completed it's work. When this happens, the JVM will fall back to a stop the world garbage collection mode. 

For efficiency's sake, the JVM attempts to start the garbage collection process as late as it can get away with. CMS tracks the growth in heapsize and attempts to time it's collection so that the collection ends right before the tenured generation fills up. Sometimes Java is wrong - such as if during the collection process, object tenuring rate increases dramatically.

You'll know if you're experiencing this failure mode if you see the words [CMS (concurrent mode failure): in your GC log.

If this failure mode is bugging you, simply set - XX:CMSInitiatingOccupancyFraction to a conservative value. This will tell the JVM to start garbage collection earlier, and thus, not run out of space so much.


2. Promotion Failure Due to Fragmentation:

This is the other biggie. If you see ParNew (promotion failed) in your GC log, you're experiencing this.

I'm gonna steal more content from Todd Lipcon and this blog post again:http://www.cloudera.com/blog/201...

This failure mode is a little bit more complicated. Recall that the CMS collector does not relocate objects, but simply tracks all of the separate areas of free space in the heap. As a thought experiment, imagine that I allocate 1 million objects, each 1KB, for a total usage of 1GB in a heap that is exactly 1GB. Then I free every odd-numbered object, so I have 500MB live. However, the free space will be solely made up of 1KB chunks. If I need to allocate a 2KB object, there is nowhere to put it, even though I ostensibly have 500MB of space free. This is termed memory fragmentation. No matter how early I ask the CMS collector to start, since it does not relocate objects, it cannot solve this problem!

When this problem occurs, the collector again falls back to the copying collector, which is able to compact all the objects and free up space.

Solutions:

Dealing with this failure mode is more difficult. As answerers have mentioned above, try to do things in a way that don't create garbage in the tenured generation - the new generation is always collected by a copying collector, thus fragmentation doesn't occur in the young generation.

The tenured generation makes the assumption that objects that are allocated together die together, but if we violate this tenet, fragmentation becomes a big problem. You can get around this by manually allocating memory in a way that objects next to eachother die at the same time, and can thus be collected together.

Todd Lipcon gives an awesome writeup of this approach with a Local Allocation Buffer in this blog post. http://www.cloudera.com/blog/201...

Another possible solution I'm working on at Cloudera is to move the most memory hungry elements of the application off-heap, either through the usage of DirectByteBuffers or via JNI. A slab allocation model similar to MemCached can be used in order to trade space efficiency for fragmentation overhead. 

You may either choose to manage memory manually, and copy stuff on heap when necessary, or wrap references to the external code in phantom references, and use a reference queue to keep track of which references have been garbage collected, and then free them using some form of free(). If you just need a cache, BigMemory provides a commercial, off the shelf solution. 

You're also free to rip out the allocator and cache I implemented inhttps://issues.apache.org/jira/b....

Obviously, both approaches employed by Todd and I are engineering intensive, and not simple to implement. But if GC tuning fails, and if you really want to minimize pauses with your Java/Scala/etc apps, then you might want to experiment with these approaches.