1. Get started with SCSS
http://alistapart.com/article/getting-started-with-sass
Saturday, August 29, 2015
Tuesday, August 18, 2015
What's the best way to avoid garbage collector pauses with big heaps in Java (Question excerpt from Quora)
Answered by Li Pi
https://www.quora.com/Whats-the-best-way-to-avoid-garbage-collector-pauses-with-big-heaps-in-Java
If the GC (Concurrent-Mark-Sweep) (see How does garbage collection work in the JVM?) is operating as expected, then the stop the world pauses should not be significant. CMS is designed to have as few stop the world pauses as possible. The Parallel New Collector will stop the world, but as long as your young generation is reasonably sized, you should be fine.
To recap, CMS operates by first stopping the world, then initially markingthe root nodes, then it concurrently proceeds to trace through the rest of the objects. Memory is freed at the end of this process, but objects are never moved around, making fragmentation problematic.
However, there exist a few failure modes where CMS will be forced the stop the world for a significant amount of time:
1. Concurrent Mode Failure - This occurs when the tenured generation fills up before CMS has completed it's work. When this happens, the JVM will fall back to a stop the world garbage collection mode.
For efficiency's sake, the JVM attempts to start the garbage collection process as late as it can get away with. CMS tracks the growth in heapsize and attempts to time it's collection so that the collection ends right before the tenured generation fills up. Sometimes Java is wrong - such as if during the collection process, object tenuring rate increases dramatically.
You'll know if you're experiencing this failure mode if you see the words [CMS (concurrent mode failure): in your GC log.
If this failure mode is bugging you, simply set - XX:CMSInitiatingOccupancy
2. Promotion Failure Due to Fragmentation:
This is the other biggie. If you see ParNew (promotion failed) in your GC log, you're experiencing this.
I'm gonna steal more content from Todd Lipcon and this blog post again:http://www.cloudera.com/b
Solutions:
Dealing with this failure mode is more difficult. As answerers have mentioned above, try to do things in a way that don't create garbage in the tenured generation - the new generation is always collected by a copying collector, thus fragmentation doesn't occur in the young generation.
The tenured generation makes the assumption that objects that are allocated together die together, but if we violate this tenet, fragmentation becomes a big problem. You can get around this by manually allocating memory in a way that objects next to eachother die at the same time, and can thus be collected together.
Todd Lipcon gives an awesome writeup of this approach with a Local Allocation Buffer in this blog post. http://www.cloudera.com/b
Another possible solution I'm working on at Cloudera is to move the most memory hungry elements of the application off-heap, either through the usage of DirectByteBuffers or via JNI. A slab allocation model similar to MemCached can be used in order to trade space efficiency for fragmentation overhead.
You may either choose to manage memory manually, and copy stuff on heap when necessary, or wrap references to the external code in phantom references, and use a reference queue to keep track of which references have been garbage collected, and then free them using some form of free(). If you just need a cache, BigMemory provides a commercial, off the shelf solution.
You're also free to rip out the allocator and cache I implemented inhttps://issues.apache.org
Obviously, both approaches employed by Todd and I are engineering intensive, and not simple to implement. But if GC tuning fails, and if you really want to minimize pauses with your Java/Scala/etc apps, then you might want to experiment with these approaches.
https://www.quora.com/Whats-the-best-way-to-avoid-garbage-collector-pauses-with-big-heaps-in-Java
If the GC (Concurrent-Mark-Sweep) (see How does garbage collection work in the JVM?) is operating as expected, then the stop the world pauses should not be significant. CMS is designed to have as few stop the world pauses as possible. The Parallel New Collector will stop the world, but as long as your young generation is reasonably sized, you should be fine.
To recap, CMS operates by first stopping the world, then initially markingthe root nodes, then it concurrently proceeds to trace through the rest of the objects. Memory is freed at the end of this process, but objects are never moved around, making fragmentation problematic.
However, there exist a few failure modes where CMS will be forced the stop the world for a significant amount of time:
1. Concurrent Mode Failure - This occurs when the tenured generation fills up before CMS has completed it's work. When this happens, the JVM will fall back to a stop the world garbage collection mode.
For efficiency's sake, the JVM attempts to start the garbage collection process as late as it can get away with. CMS tracks the growth in heapsize and attempts to time it's collection so that the collection ends right before the tenured generation fills up. Sometimes Java is wrong - such as if during the collection process, object tenuring rate increases dramatically.
You'll know if you're experiencing this failure mode if you see the words [CMS (concurrent mode failure): in your GC log.
If this failure mode is bugging you, simply set - XX:CMSInitiatingOccupancy
2. Promotion Failure Due to Fragmentation:
This is the other biggie. If you see ParNew (promotion failed) in your GC log, you're experiencing this.
I'm gonna steal more content from Todd Lipcon and this blog post again:http://www.cloudera.com/b
This failure mode is a little bit more complicated. Recall that the CMS collector does not relocate objects, but simply tracks all of the separate areas of free space in the heap. As a thought experiment, imagine that I allocate 1 million objects, each 1KB, for a total usage of 1GB in a heap that is exactly 1GB. Then I free every odd-numbered object, so I have 500MB live. However, the free space will be solely made up of 1KB chunks. If I need to allocate a 2KB object, there is nowhere to put it, even though I ostensibly have 500MB of space free. This is termed memory fragmentation. No matter how early I ask the CMS collector to start, since it does not relocate objects, it cannot solve this problem!
When this problem occurs, the collector again falls back to the copying collector, which is able to compact all the objects and free up space.
Solutions:
Dealing with this failure mode is more difficult. As answerers have mentioned above, try to do things in a way that don't create garbage in the tenured generation - the new generation is always collected by a copying collector, thus fragmentation doesn't occur in the young generation.
The tenured generation makes the assumption that objects that are allocated together die together, but if we violate this tenet, fragmentation becomes a big problem. You can get around this by manually allocating memory in a way that objects next to eachother die at the same time, and can thus be collected together.
Todd Lipcon gives an awesome writeup of this approach with a Local Allocation Buffer in this blog post. http://www.cloudera.com/b
Another possible solution I'm working on at Cloudera is to move the most memory hungry elements of the application off-heap, either through the usage of DirectByteBuffers or via JNI. A slab allocation model similar to MemCached can be used in order to trade space efficiency for fragmentation overhead.
You may either choose to manage memory manually, and copy stuff on heap when necessary, or wrap references to the external code in phantom references, and use a reference queue to keep track of which references have been garbage collected, and then free them using some form of free(). If you just need a cache, BigMemory provides a commercial, off the shelf solution.
You're also free to rip out the allocator and cache I implemented inhttps://issues.apache.org
Obviously, both approaches employed by Todd and I are engineering intensive, and not simple to implement. But if GC tuning fails, and if you really want to minimize pauses with your Java/Scala/etc apps, then you might want to experiment with these approaches.
How does garbage collection work in JVM (Question excerpt from Quora)
Answered by Li Pi
https://www.quora.com/How-does-garbage-collection-work-in-the-JVM
This actually varies depending on the JVM implementation, but I'm assuming you're talking about Oracle (Sun) Version 6. Java itself does not specify a particular method of garbage collection.
The JVM uses a form of garbage collector called a tracing collector, which essentially operates by first stopping the world, marking all root objects, or objects that are referenced directly by running threads, and following references, marking each object it hits along the way.
Java 6 implements something called a generational garbage collector—based upon the generational hypothesis assumption, which states that the majority of objects that are created are quickly discarded, and that objects that are not quickly collected are likely to be around for some time.
Based upon these assumptions, Java therefore partitions objects into two different generations, and then operates differently upon them.
Visually, the generations look like this:
(Not quite to scale)
Young Generation: This is where objects start out. It has two subgenerations:
Tenured Generation: Older objects with longer lifetimes end up here.
Java is smart enough to apply different garbage collection methods to each generation. The young generation is handled using a tracing, copyingcollector called the Parallel New Collector. This collector stops the world, but because the young generation is generally small, the pause is short.
For the young generation:
When Eden fills up, the garbage collector stops the world, then traces through the objects in the young generation, starting with those referenced immediately by a running thread.
Those that are marked or "alive" are copied over to the empty survivor space. This survivor space is then marked as "live", and Eden, along with the other survivor space, is marked as empty. This has the side effect of compacting all the objects into a single survivor space, allowing for rather efficient memory usage. If an object has been copied between the two survivor places a certain amount of times, its designated as tenured, and moved to the tenured section.
Eden will now be overwritten by new objects, and the next garbage collection cycle will proceed to use the other survivor space.
This usage of a copying collector for the young generation is fast because the vast majority of objects are very quickly destroyed, and generally, very few objects must be moved around.
For the tenured generation:
I'm going to steal this section from this blog post by Todd Lipcon,http://www.cloudera.com/b
Tenured Generation – Concurrent Mark-Sweep
Every time the parallel new collector runs, it will tenure some objects into the tenured generation. So, of course, the old generation will eventually fill up, and we need a strategy for collecting it as well. The Concurrent-Mark-Sweep collector (CMS) is responsible for clearing dead objects in this generation.
The CMS collector operates in a series of phases. Some phases stop the world, and others run concurrently with the Java application. The major phases are:
The important things to note here are:
https://www.quora.com/How-does-garbage-collection-work-in-the-JVM
This actually varies depending on the JVM implementation, but I'm assuming you're talking about Oracle (Sun) Version 6. Java itself does not specify a particular method of garbage collection.
The JVM uses a form of garbage collector called a tracing collector, which essentially operates by first stopping the world, marking all root objects, or objects that are referenced directly by running threads, and following references, marking each object it hits along the way.
Java 6 implements something called a generational garbage collector—based upon the generational hypothesis assumption, which states that the majority of objects that are created are quickly discarded, and that objects that are not quickly collected are likely to be around for some time.
Based upon these assumptions, Java therefore partitions objects into two different generations, and then operates differently upon them.
Visually, the generations look like this:
(Not quite to scale)
Young Generation: This is where objects start out. It has two subgenerations:
- Eden - Objects start out here.
- Survivor - Objects that survive Eden end up here. There are two of these, and only one is in use at any given time. One is designated as empty, and the other as live. This switched every GC cycle.
Tenured Generation: Older objects with longer lifetimes end up here.
Java is smart enough to apply different garbage collection methods to each generation. The young generation is handled using a tracing, copyingcollector called the Parallel New Collector. This collector stops the world, but because the young generation is generally small, the pause is short.
For the young generation:
When Eden fills up, the garbage collector stops the world, then traces through the objects in the young generation, starting with those referenced immediately by a running thread.
Those that are marked or "alive" are copied over to the empty survivor space. This survivor space is then marked as "live", and Eden, along with the other survivor space, is marked as empty. This has the side effect of compacting all the objects into a single survivor space, allowing for rather efficient memory usage. If an object has been copied between the two survivor places a certain amount of times, its designated as tenured, and moved to the tenured section.
Eden will now be overwritten by new objects, and the next garbage collection cycle will proceed to use the other survivor space.
This usage of a copying collector for the young generation is fast because the vast majority of objects are very quickly destroyed, and generally, very few objects must be moved around.
For the tenured generation:
I'm going to steal this section from this blog post by Todd Lipcon,http://www.cloudera.com/b
Tenured Generation – Concurrent Mark-Sweep
Every time the parallel new collector runs, it will tenure some objects into the tenured generation. So, of course, the old generation will eventually fill up, and we need a strategy for collecting it as well. The Concurrent-Mark-Sweep collector (CMS) is responsible for clearing dead objects in this generation.
The CMS collector operates in a series of phases. Some phases stop the world, and others run concurrently with the Java application. The major phases are:
- initial-mark (stops the world). In this phase, the CMS collector places amark on the rootobjects. A root object is something directly referenced from a live Thread – for example, the local variables in use by that thread. This phase is short because the number of roots is very small.
- concurrent-mark (concurrent). The collector now follows every pointer starting from the root objects until it has marked all live objects in the system.
- remark (stops the world). Since objects might have had references changed, and new objects might have been created during concurrent-mark, we need to go back and take those into account in this phase. This is short because a special data structure allows us to only inspect those objects that were modified during the prior phase.
- concurrent-sweep (concurrent). Now, we proceed through all objects in the heap. Any object without a mark is collected and considered free space. New objects allocated during this time are marked as they are created so that they aren’t accidentally collected.
The important things to note here are:
- The stop-the-world phases are made to be very short. The long work of scanning the whole heap and sweeping up the dead objects happens concurrently.
- This collector does not relocate the live objects, so free space can be spread in different chunks throughout the heap. We’ll come back to this later!
Why do most software engineers use a MacBook instead of a Windows or a Linux laptop (Question excerpt from Quora)
---- Answered by Eric Zheng
https://www.quora.com/Question-That-Contains-Assumptions-1/Why-do-most-software-engineers-use-a-MacBook-instead-of-a-Windows-or-a-Linux-laptop
Several big things coincidently happened around the same time in late last decade:
Combined together, they are why MacBook is so much more popular today than a decade ago (circa 2005):
1. Mobile app
The population of iOS and Android app developers is huge. It makes a lot of sense to develop iOS and Android apps on Mac. iPhone and Android were born around 2007/2008.
2. Amazon Web Services
Public cloud, especially AWS, made it much easier to provision servers. However, AWS, the leading and dominant public cloud provider launched in2006, didn't support windows in the first a couple years. It added Windows support in EC2 in 2008 as a beta feature. In the following a couple years, as far as I remember, the developer community was still staying away from running Windows on AWS because the glitches here and there. Therefore, in order to take advantage of the cloud computing, many people had no choice but start to pick up Linux.
3. Startup
Tech startups started taking off soon after the 2008-2009 recession. One of the reasons was that AWS made it much easier to do startup. Startup prefer use AWS for cost saving and other cloud computing benefits. Startup attracted many best programmers, for many reasons including the potential huge financial return. Because of having joined startups, many people started to pick up Linux. Also, good programmers make their work platform better. An example in the history was that VB/Delphi/VC++ programmers on Windows 95/98, in their side project, created a lot of great softwares to scratch their own itches, such as to download/upload files (remember those FTP clients and HTTP downloaders?), play MP3 and video (WinAmp, ...), editing text file, etc.. Similar thing happened to OS X platform in the last several years. That is the power of developer ecosystem, which helped make Windows successful.
4. OS X on x86
In 2005, Apple announced to change OS X to x86 architecture. The transition started from 2006, when Apple released the first x86 MacBook, and completed in 2009 when OS X 10.6 was released which only supported x86.
This switch has two major impacts:
First, Mac hardware changing to x86 allowed us to install Windows on MacBook, using Boot Camp (released in 2006). That made it a lot more comfortable for people to buy a MacBook as their only personal computer. Because prior to that, although many people loved the build quality and form factor of a MacBook, they were afraid of the software issues. For example, there were a lot of websites using ActiveX which weren't very well supported on Mac OS X. Now with dual systems, they have a way out: when necessary, they can boot into the Windows.
Second, now lots of the tools that work on Linux can be much more easily ported to Mac OS X, because the tools are mostly written for x86 architecture and now OS X is also x86. Like many others said, now OS X is a much nicer Linux workstation.
5. Git and GitHub
Because git was created by the creator of Linux, git works really well on Linux. Because of git, GitHub was possible and got popular. Because of git (2005) and GitHub (2008), it became much easier to do open source, compared to the days when we only had SourceForge, CodePlex and Google Code. When it's easy to do open source projects, more open source tools and lib are created, which leads to more applications on the platform (Linux/Mac OS).
p.s. I previously partially misunderstood what the asker meant. The asker was saying that the hardware MacBook is the most popular. S/He didn't say Mac OS X was the most popular. The asker seemed to have said that many MacBook hardware is running Windows.
Having admitted the misunderstanding, I would like to keep my above answer, since it still explains my view of why MacBook (running Mac OS X) has become very popular among developers.
Personally I don't see many people running Windows as the primary OS on their MacBook.
https://www.quora.com/Question-That-Contains-Assumptions-1/Why-do-most-software-engineers-use-a-MacBook-instead-of-a-Windows-or-a-Linux-laptop
Several big things coincidently happened around the same time in late last decade:
- iOS and Android - 2007-2008
- Amazon Web Services - 2006
- x86 MacBook and Mac OS - 2006
- Startup booming - post 2008-2009 recession
- Git and GitHub - 2005 (git) and 2008 (github)
Combined together, they are why MacBook is so much more popular today than a decade ago (circa 2005):
1. Mobile app
The population of iOS and Android app developers is huge. It makes a lot of sense to develop iOS and Android apps on Mac. iPhone and Android were born around 2007/2008.
2. Amazon Web Services
Public cloud, especially AWS, made it much easier to provision servers. However, AWS, the leading and dominant public cloud provider launched in2006, didn't support windows in the first a couple years. It added Windows support in EC2 in 2008 as a beta feature. In the following a couple years, as far as I remember, the developer community was still staying away from running Windows on AWS because the glitches here and there. Therefore, in order to take advantage of the cloud computing, many people had no choice but start to pick up Linux.
3. Startup
Tech startups started taking off soon after the 2008-2009 recession. One of the reasons was that AWS made it much easier to do startup. Startup prefer use AWS for cost saving and other cloud computing benefits. Startup attracted many best programmers, for many reasons including the potential huge financial return. Because of having joined startups, many people started to pick up Linux. Also, good programmers make their work platform better. An example in the history was that VB/Delphi/VC++ programmers on Windows 95/98, in their side project, created a lot of great softwares to scratch their own itches, such as to download/upload files (remember those FTP clients and HTTP downloaders?), play MP3 and video (WinAmp, ...), editing text file, etc.. Similar thing happened to OS X platform in the last several years. That is the power of developer ecosystem, which helped make Windows successful.
4. OS X on x86
In 2005, Apple announced to change OS X to x86 architecture. The transition started from 2006, when Apple released the first x86 MacBook, and completed in 2009 when OS X 10.6 was released which only supported x86.
This switch has two major impacts:
First, Mac hardware changing to x86 allowed us to install Windows on MacBook, using Boot Camp (released in 2006). That made it a lot more comfortable for people to buy a MacBook as their only personal computer. Because prior to that, although many people loved the build quality and form factor of a MacBook, they were afraid of the software issues. For example, there were a lot of websites using ActiveX which weren't very well supported on Mac OS X. Now with dual systems, they have a way out: when necessary, they can boot into the Windows.
Second, now lots of the tools that work on Linux can be much more easily ported to Mac OS X, because the tools are mostly written for x86 architecture and now OS X is also x86. Like many others said, now OS X is a much nicer Linux workstation.
5. Git and GitHub
Because git was created by the creator of Linux, git works really well on Linux. Because of git, GitHub was possible and got popular. Because of git (2005) and GitHub (2008), it became much easier to do open source, compared to the days when we only had SourceForge, CodePlex and Google Code. When it's easy to do open source projects, more open source tools and lib are created, which leads to more applications on the platform (Linux/Mac OS).
p.s. I previously partially misunderstood what the asker meant. The asker was saying that the hardware MacBook is the most popular. S/He didn't say Mac OS X was the most popular. The asker seemed to have said that many MacBook hardware is running Windows.
Having admitted the misunderstanding, I would like to keep my above answer, since it still explains my view of why MacBook (running Mac OS X) has become very popular among developers.
Personally I don't see many people running Windows as the primary OS on their MacBook.
Monday, August 17, 2015
The Netflix Tech Blog: Making Netflix.com Faster
The Netflix Tech Blog: Making Netflix.com Faster: by Kristofer Baxter Simply put, performance matters. We know members want to immediately start browsing or watching their favorite content ...
Subscribe to:
Posts (Atom)