tag:blogger.com,1999:blog-340491302024-03-18T08:24:53.182+01:00The Tapir's TaleProgramming, Philosophy, and the Mind.Anders Janmyrhttp://www.blogger.com/profile/16081537018884349622noreply@blogger.comBlogger147125tag:blogger.com,1999:blog-34049130.post-31415227389344892512023-02-19T12:11:00.001+01:002023-02-19T12:11:51.429+01:00Appropriate Specificity of a Compliment
<p>What is the appropriate level of a compliment? You would think that the
more generic the better since that includes a larger number, but clearly
"You are most beautiful woman", beats "You are the most beautiful mammal". What's going on?</p>
<p>Becoming more specific is not better either. "You are the most beautiful woman in your
forties" does not sound good either, especially as a personal comment. It might
work as a more impersonal compliment in a listing of "sexy women over 40", but
clearly not as a personal compliment. Getting more specific clearly makes it
worse. "You are the most beautiful women over 40 with red hair", is also worse.</p>
<p>Let's put this this on a scale.</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhx4ciS1g-BiRrMHCeRJ6T2_Z84kP77rdxGMQg8bpmlkM3L69U9t0vpzgCvF2B3vMMIy4MEWmbGSm6GdvYkgCj6hAX5g5PesDPcfaBd5-OdQrsY782aaoHDSD4wb7lUGw3dnCGqEf5WRaN13xHbdSVZ9ABrZx2A7hEMCmkHpk1v7U6zJFE6eQ/s750/compliments_women.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="400" data-original-width="750" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhx4ciS1g-BiRrMHCeRJ6T2_Z84kP77rdxGMQg8bpmlkM3L69U9t0vpzgCvF2B3vMMIy4MEWmbGSm6GdvYkgCj6hAX5g5PesDPcfaBd5-OdQrsY782aaoHDSD4wb7lUGw3dnCGqEf5WRaN13xHbdSVZ9ABrZx2A7hEMCmkHpk1v7U6zJFE6eQ/s400/compliments_women.jpg"/></a></div>
<p>Combining the compliment, specific and generic doesn't help either. It also
makes it worse. "You are the most beautiful red haired mammal in your forties" doesn't come out right.</p>
<h3>Other categories</h3>
<p>There are clearly differences in what category we vary too. Varying time and
geographical area is does not follow a normal curve at all.</p>
<h3>Geographical Area</h3>
<p>"You are the most beautiful woman in the world", is better than "You are the most beautiful
woman in Sweden with is clearly better than "You are the
most beautiful woman in this household". This seems to follow a more proportional
distribution where larger is better.</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgbua2D-HSIhRodRYq_CyOJQmxbqLDEKCq5o6hDgENcFo2knn2wpGvVg1peeeNBhVzlqTykL4kf7MJ7Yv6Y-oq2xIFCP3YftnnvHKfa1TST4mYxw2ccJukZFt_SLALdl_tipM6AesY_-mq0RKZ82rWKgdsLeWNTuXKu6elLm8H6XcUonaa5rg/s750/compliment_area.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="400" data-original-width="750" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgbua2D-HSIhRodRYq_CyOJQmxbqLDEKCq5o6hDgENcFo2knn2wpGvVg1peeeNBhVzlqTykL4kf7MJ7Yv6Y-oq2xIFCP3YftnnvHKfa1TST4mYxw2ccJukZFt_SLALdl_tipM6AesY_-mq0RKZ82rWKgdsLeWNTuXKu6elLm8H6XcUonaa5rg/s400/compliment_area.jpg"/></a></div>
<h3>Temporal</h3>
<p>"You are the most beautiful woman I've ever seen", is better than "You are the most beautiful
woman I've seen this week. At first, it have the same proportionality as area,
but when we come to the far end of time, <strong>now</strong>, we see a little twist in
appreciation. It is definitely better to be "most beautiful right now" than "this
week".</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj_aTAj7ByIfkYT_kXl8DQg5E1o1uqr67yRXAe39G3f2WZHlFaZED7VaF4ekuzWNdbZZiiFdfI5Q9mNReHfL2L0AJxJcZUInyLbKkid9NVMEKLsr5seIzJESjULTEN6hWBlKkmorfENXxe-uRMT1kta9guEwGFFpjzQ3JZFCzKuNiPJ2FRbyQ/s750/comliment_in_time.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="400" data-original-width="750" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj_aTAj7ByIfkYT_kXl8DQg5E1o1uqr67yRXAe39G3f2WZHlFaZED7VaF4ekuzWNdbZZiiFdfI5Q9mNReHfL2L0AJxJcZUInyLbKkid9NVMEKLsr5seIzJESjULTEN6hWBlKkmorfENXxe-uRMT1kta9guEwGFFpjzQ3JZFCzKuNiPJ2FRbyQ/s400/comliment_in_time.jpg"/></a></div>
<p>The only conclusion to draw from this is that compliments are not easy. :)</p>
Anders Janmyrhttp://www.blogger.com/profile/16081537018884349622noreply@blogger.com0tag:blogger.com,1999:blog-34049130.post-89199384023884176552022-10-26T22:16:00.002+02:002022-10-27T08:13:47.001+02:00Overpopulation
<p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhDItYT48fHZ9iyFmJMJrBZWt-55MZj24-ymYnT_GuMJS6BCOAHsOvXx819J89KKQmZ5rz9i9l7ZE_TwBwwlouQHXUXVT5u5O8wh4kYvbb9PAHu_UQsWpm1q_Mu4dMQq3kbervgXFFIdgLomH9PYB3EgqoWpFDe5UL897nPFmXMNkYr7jaipw/s1920/overpopulation.001.jpeg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="1080" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhDItYT48fHZ9iyFmJMJrBZWt-55MZj24-ymYnT_GuMJS6BCOAHsOvXx819J89KKQmZ5rz9i9l7ZE_TwBwwlouQHXUXVT5u5O8wh4kYvbb9PAHu_UQsWpm1q_Mu4dMQq3kbervgXFFIdgLomH9PYB3EgqoWpFDe5UL897nPFmXMNkYr7jaipw/s400/overpopulation.001.jpeg"/></a></div>
</p>
<p>COP27 is coming up. One topic that wasn't negotiated in COP26 was
population. Currently, global population
is increasing with 80 million people every year.</p>
<p><a href="https://twitter.com/elonmusk/status/1563020169160851456?lang=en">Smart</a> and <a href="https://twitter.com/jordanbpeterson/status/1477890383799414785">influential</a> people argues that aging population is a way bigger problem than over-population
since new generations will not be able to support their elders and keep up
progress.</p>
<p>I believe they are mistaken in two ways:</p>
<ul>
<li>More people requires more resources and I don't believe the world has enough.</li>
<li>Automation renders a lot of people unable to find sustainable jobs.</li>
</ul>
<p>Kimberly Nicholas, PhD in Sustainability Science,
<a href="(https://www.amazon.com/Under-Sky-We-Make-Warming/dp/0593328175?tag=thtasta-20)">argues</a>
that one of the four most impactful things we in the over-consuming world can
do o lower emissions is to have less children.</p>
<p>This should not only apply to the over-consuming world, since people in
developing countries tend to migrate to developed countries and hence become
high emitters like the rest of us.</p>
<p>
<h2>Kaya</h2>
<div class="separator" style="clear: both;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgPVk5RAhCzSW_9LbYSIzluYcTOFhUIGdKm-P2XvKdQw8nl4Qp_ENkugKJL-kmyPcSw5m-d4JBUo34AsQvYnwnzmkQ49XMDLvYj96r_nVEMzjF2Zw6PqY0EJple_u07VcXgm_-8twPG1JTpLTxhOP90EQaj1C7-5BKD3pIhoSwJ6b2G0KsQ7A/s1920/overpopulation.002.jpeg" style="display: block; padding: 1em 0; text-align: center; clear: right; float: right;"><img alt="" border="0" width="320" data-original-height="1080" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgPVk5RAhCzSW_9LbYSIzluYcTOFhUIGdKm-P2XvKdQw8nl4Qp_ENkugKJL-kmyPcSw5m-d4JBUo34AsQvYnwnzmkQ49XMDLvYj96r_nVEMzjF2Zw6PqY0EJple_u07VcXgm_-8twPG1JTpLTxhOP90EQaj1C7-5BKD3pIhoSwJ6b2G0KsQ7A/s320/overpopulation.002.jpeg"/></a></div>
</p>
<p>There is a simple formula for calculating global carbon emissions in economic
terms. It is called the <a href="http://climatemodels.uchicago.edu/kaya/">Kaya Identity Scenario Prognosticator</a>
and it is named after it's inventor, Yoichi Kaya, a Japanese Energy Economist.</p>
<p>Lowering any of the four categories, will lower our Carbon emissions. Getting
any single one of them down to zero will completely stop our emissions.</p>
<div class="separator" style="clear: both;"></div>
<p>
<h2>Increasing and Decreasing</h2>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhPlKyXG6KyYdciEnHkBH2bl36FkM3cwGZGXZXCMvOy6dj6NRoF6tjveKflG8Vc-v-d7SQ_RqmnYrBfgpt5sruoU3ytvV9FdywqKMLp6OsxOAZ3ZDXN27OEBk1IitoLRDj7p-iyvdUL_15JhcwQxvJNXegVFRAuAmEqc5vWDTqmdpHGCrgjqw/s1920/overpopulation.003.jpeg" style="display: block; padding: 1em 0; text-align: center; clear: left; float: left;"><img alt="" border="0" width="320" data-original-height="1080" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhPlKyXG6KyYdciEnHkBH2bl36FkM3cwGZGXZXCMvOy6dj6NRoF6tjveKflG8Vc-v-d7SQ_RqmnYrBfgpt5sruoU3ytvV9FdywqKMLp6OsxOAZ3ZDXN27OEBk1IitoLRDj7p-iyvdUL_15JhcwQxvJNXegVFRAuAmEqc5vWDTqmdpHGCrgjqw/s320/overpopulation.003.jpeg"/></a></div>
</p>
<p>Currently, two of the categories or decreasing while two are increasing. It's
not hard to understand that this is not an optimal solution.</p>
<div class="separator" style="clear: both;"></div>
<p>
<h2>Carbon Emissions per Unit of Energy</h2>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgqL_OsyRD6Yb-i102ezcAkUJlEsgoHyeKPUvhJJucLFg4KTdMshF0Tv0yAK-0qlS91b93F-lDeEDZzZ3GoE_sWAlvUT8MTtcCkF4IPvwvTeCNax8GLmBlrZEgaoyLljWXiTwuaOXFdG7weJSTPaIVLqm2HWgOeOo6MiroPzamG3bg5mBaPBw/s1920/overpopulation.004.jpeg" style="display: block; padding: 1em 0; text-align: center; clear: right; float: right;"><img alt="" border="0" width="320" data-original-height="1080" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgqL_OsyRD6Yb-i102ezcAkUJlEsgoHyeKPUvhJJucLFg4KTdMshF0Tv0yAK-0qlS91b93F-lDeEDZzZ3GoE_sWAlvUT8MTtcCkF4IPvwvTeCNax8GLmBlrZEgaoyLljWXiTwuaOXFdG7weJSTPaIVLqm2HWgOeOo6MiroPzamG3bg5mBaPBw/s320/overpopulation.004.jpeg"/></a></div>
</p>
<p>Furthest to the right, we have Carbon Emissions per Unit of Energy. This is the
one most of the fuzz is about. How can we lower the amount of carbon emitted
per amount of energy. The answer is simple, but the implementation is not. We
need to replace fossil energy with fossil free energy, solar, wind, biofuel,
and nuclear. The only controversy is nuclear which I will not dwell on since
I already wrote an article on how
<a href="https://anders.janmyr.com/2021/04/global-warming-time-for-nuclear-option.html">I think we should go all in on Nuclear</a>.
But, thankfully not every country are as squeamish about nuclear as we Swedes and the
Germans are. France is leading the charge and China is planning 150 new nuclear plants.</p>
<div class="separator" style="clear: both;"></div>
<p>
<h2>Energy use per GDP, efficiency</h2>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgK8vO6TlxEDOHDxxpw5HtzmNJ9jJSpoi31QZ3Xs5KOd3qltE8YIiFDG4TTR3owfYfTymU2ykSuNAuU4HjFqd0qmhBFZQgEfnQnQI9VvD9YZZM68xk1clPFFBru8cQOFsKyGKjHqWRRw4K-GXcH9BVdK2_6TDZ_S0aLokGeYHnnkXBbGFNQGA/s1920/overpopulation.005.jpeg" style="display: block; padding: 1em 0; text-align: center; clear: left; float: left;"><img alt="" border="0" width="320" data-original-height="1080" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgK8vO6TlxEDOHDxxpw5HtzmNJ9jJSpoi31QZ3Xs5KOd3qltE8YIiFDG4TTR3owfYfTymU2ykSuNAuU4HjFqd0qmhBFZQgEfnQnQI9VvD9YZZM68xk1clPFFBru8cQOFsKyGKjHqWRRw4K-GXcH9BVdK2_6TDZ_S0aLokGeYHnnkXBbGFNQGA/s320/overpopulation.005.jpeg"/></a></div>
</p>
<p>This is how much energy we get for every dollar we spend. This is also called
efficiency and it's something we are very good at. We keep making better
motors, batteries, production methods, etc. There is nothing controversial in this area. We would do this anyway since it is good economy to be efficient.</p>
<div class="separator" style="clear: both;"></div>
<p>
<h2>Economic Growth</h2>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgiU5iSr0AUn_TfVZbgk8IQVw5fGM-0_3YKS21XQ8gnXiRV6C21lh3KXPqEeIPeOSKpFcamVi1Bx0W3sPA50OQ5_hdVTfY7P8Aj3DzAsvLfdYRBHoh_VjHrisbfpHIQHy8OQ4fKljfY2jZzLvOFShmZZD0DJJOll5_qxf-0eZs1_iTPn2xl1Q/s1920/overpopulation.006.jpeg" style="display: block; padding: 1em 0; text-align: center; clear: right; float: right;"><img alt="" border="0" width="320" data-original-height="1080" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgiU5iSr0AUn_TfVZbgk8IQVw5fGM-0_3YKS21XQ8gnXiRV6C21lh3KXPqEeIPeOSKpFcamVi1Bx0W3sPA50OQ5_hdVTfY7P8Aj3DzAsvLfdYRBHoh_VjHrisbfpHIQHy8OQ4fKljfY2jZzLvOFShmZZD0DJJOll5_qxf-0eZs1_iTPn2xl1Q/s320/overpopulation.006.jpeg"/></a></div>
</p>
<p>This is where the controversy starts to grow. Economists wants growth to be
high since it will allow TODO, while many ecologists wants to stop growth (at
least in the developed world) since they don't think it is sustainable. I agree
that in the developed world, we certainly have room to lower our consumption,
but I'm not sure what it would do to our economy since so many things are based
on this ever increasing growth. But, most of the world just want to reach
a level that doesn't force them to struggle every day. This is a tough nut to
crack.</p>
<p>Economic growth also increases with migration. When people migrate to higher
consumption countries, they increase their consumption.</p>
<div class="separator" style="clear: both;"></div>
<p>
<h2>Population</h2>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh68VfqH9AfrL2QvPeRBkbw5f6M2rrodAIFGmLHc1TvXckuwdGPsy2rhWBpOz7JJ7e375X6LeOuMfqHVxKf1du5_GQyzJGZ5JkUVANy_lOrtwBXa1ICP3JeTcDCJuspI5s7YjTWc6mKdDdorJdmHXwUz3yf7wdouqPbVNcBBkVVo-zVV1Mojw/s1920/overpopulation.007.jpeg" style="display: block; padding: 1em 0; text-align: center; clear: left; float: left;"><img alt="" border="0" width="320" data-original-height="1080" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh68VfqH9AfrL2QvPeRBkbw5f6M2rrodAIFGmLHc1TvXckuwdGPsy2rhWBpOz7JJ7e375X6LeOuMfqHVxKf1du5_GQyzJGZ5JkUVANy_lOrtwBXa1ICP3JeTcDCJuspI5s7YjTWc6mKdDdorJdmHXwUz3yf7wdouqPbVNcBBkVVo-zVV1Mojw/s320/overpopulation.007.jpeg"/></a></div>
</p>
<p>Finally, population. It's increasing by 80 million people yearly, and it's
expected to peak somewhere around 10 billion people. This is an immense problem
that it seems that no one wants to talk about! It's an ecosystem out of
balance.</p>
<div class="separator" style="clear: both;"></div>
<p><h2>Population vs Warming</h2>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEisIc_03mG3Ho1pROvnt8qzdRDrIBzm8JSREfUwiYjn0KqNnPojxmwwdYqeeu4iNXUmlnBOSvchGjnZpA0rCDSawHjh5axtH1GCQW_DtokfZuhNaH7Oc1lCAp_EzH0VdC_iGC5vdt9E2OsP_Opfc79HyhC3gTkc3mbrg3qEaDJmGj2nOIPI-w/s1920/overpopulation.008.jpeg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="1080" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEisIc_03mG3Ho1pROvnt8qzdRDrIBzm8JSREfUwiYjn0KqNnPojxmwwdYqeeu4iNXUmlnBOSvchGjnZpA0rCDSawHjh5axtH1GCQW_DtokfZuhNaH7Oc1lCAp_EzH0VdC_iGC5vdt9E2OsP_Opfc79HyhC3gTkc3mbrg3qEaDJmGj2nOIPI-w/s400/overpopulation.008.jpeg"/></a></div>
</p>
<p>It is eerie how much population correlates with temperature.</p>
<div class="separator" style="clear: both;"></div>
<p>
<h2>Global Unemployment</h2>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEix_mL2MA4Fw9bpTc0H1Pn3rSfX_UJ-_TnEU9UJrIP9qrwQJISjI7vXbBDEOeYIPHoYt0JUQzt039EAJ72WX8g4tkQVBVHwsw26YwOiBEgLoRTlIknDIWyUAN1UtPJs29CL1k8JgutByJwEeO_Q1VxrvIwg60PvUPFSdK4Pu0Z23-U9EQappQ/s1920/overpopulation.009.jpeg" style="display: block; padding: 1em 0; text-align: center; clear: right; float: right;"><img alt="" border="0" width="320" data-original-height="1080" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEix_mL2MA4Fw9bpTc0H1Pn3rSfX_UJ-_TnEU9UJrIP9qrwQJISjI7vXbBDEOeYIPHoYt0JUQzt039EAJ72WX8g4tkQVBVHwsw26YwOiBEgLoRTlIknDIWyUAN1UtPJs29CL1k8JgutByJwEeO_Q1VxrvIwg60PvUPFSdK4Pu0Z23-U9EQappQ/s320/overpopulation.009.jpeg"/></a></div>
</p>
<p>Our increased efficiency also increases global unemployment. Yuval Noah Harari has <a href="https://ideas.ted.com/the-rise-of-the-useless-class/">predicted the rise of the useless class</a> when better automation and AI makes many people redundant.</p>
<p>
<blockquote>"people devoid of any economic, political or even artistic value, who contribute nothing to the prosperity, power and glory of society. This "useless class" will not merely be unemployed — it will be unemployable".
</blockquote>
</p>
<div class="separator" style="clear: both;"></div>
<p>
<h2>Out of Resources</h2>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjGzflyXE6aRyVqGIlE_xJavPRE-DCkD564dmmXk76cpAOnXmFE4tbbyZEBnpxjeCIbiV62rlkEy6HkJ_ng-cSg4TOP_qh1J9Gtl4P31nT-OCi1yHd_xHw55un1NpTWNKC8cXP1CMBoR8uFX0TVpwtPIyrgymMfD2gzSzWrqEEnXOZL-sHE7w/s1920/overpopulation.010.jpeg" style="display: block; padding: 1em 0; text-align: center; clear: left; float: left;"><img alt="" border="0" width="320" data-original-height="1080" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjGzflyXE6aRyVqGIlE_xJavPRE-DCkD564dmmXk76cpAOnXmFE4tbbyZEBnpxjeCIbiV62rlkEy6HkJ_ng-cSg4TOP_qh1J9Gtl4P31nT-OCi1yHd_xHw55un1NpTWNKC8cXP1CMBoR8uFX0TVpwtPIyrgymMfD2gzSzWrqEEnXOZL-sHE7w/s320/overpopulation.010.jpeg"/></a></div>
</p>
<p>More people requires more food, more transportation, more, more and more!
Earth is finite and according to "The World Counts" we are running out fast.
<a href="https://www.theworldcounts.com/challenges/planet-earth">https://www.theworldcounts.com/challenges/planet-earth</a></p>
<p>Vaclav Smil writes in his book, <a href="https://www.amazon.com/How-World-Really-Works-Science/dp/0593297067?tag=thtasta-20">How the World Really Works</a>, that agriculture without carbon emitting fertilizers can only suppport around 4 billion people.
<div class="separator" style="clear: both;"></div>
<p>
<h2>Demographic Transition Model</h2>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9CBimHd-5lKsPK3gML8IJq7q_W2CzGL3xyJr1Dn4NQugdsuJ4RGKtT7EfF_aYUvot547QaB2IdtBaW6XdZcsbzxqNOQubyewRscuJ-Rpj3s2K25K4q9i8DuOzmayH8ypbkXAPjZNn-4N0hmNkMHT9EE_EL_cMi5CPLsSMA7MnsHH5WpBLfw/s1920/overpopulation.011.jpeg" style="display: block; padding: 1em 0; text-align: center; clear: right; float: right;"><img alt="" border="0" width="320" data-original-height="1080" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9CBimHd-5lKsPK3gML8IJq7q_W2CzGL3xyJr1Dn4NQugdsuJ4RGKtT7EfF_aYUvot547QaB2IdtBaW6XdZcsbzxqNOQubyewRscuJ-Rpj3s2K25K4q9i8DuOzmayH8ypbkXAPjZNn-4N0hmNkMHT9EE_EL_cMi5CPLsSMA7MnsHH5WpBLfw/s320/overpopulation.011.jpeg"/></a></div>
</p>
<p>The Demographic Transition Model shows us that population growth is a temporary phenomena, but will the growth end in time? We are currently somewhere in phase 3.
Growth will end for one reason or another. It could be because we decide to solve it or that we run out of resources.</p>
<div class="separator" style="clear: both;"></div>
<p>
<h2>Births vs Deaths</h2>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEghQgcIfZA513hqRsF28bdxzYlIFB1HW81DntjJP7x639Tvkt8rgg6_Q7hEk3_AINEGKx9R-8bF5xry5w2srS3AUKLCOjhSX8WBCLeox4vsP5BGIHrBlLJjTi5o2uOmErve7RykKx24prPJxNmGe_fWBb67Tj8y1CVOKCVWLxiXvjnk1PpNxg/s1920/overpopulation.012.jpeg" style="display: block; padding: 1em 0; text-align: center; clear: left; float: left;"><img alt="" border="0" width="320" data-original-height="1080" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEghQgcIfZA513hqRsF28bdxzYlIFB1HW81DntjJP7x639Tvkt8rgg6_Q7hEk3_AINEGKx9R-8bF5xry5w2srS3AUKLCOjhSX8WBCLeox4vsP5BGIHrBlLJjTi5o2uOmErve7RykKx24prPJxNmGe_fWBb67Tj8y1CVOKCVWLxiXvjnk1PpNxg/s320/overpopulation.012.jpeg"/></a></div>
</p>
<p>In this graph we see that the number of births has topped out at around 140 million per year while the number of deaths is only increasing slightly. This is due to a lot lower childhood mortality and because we are living longer across the globe.</p>
<div class="separator" style="clear: both;"></div>
<p>
<h2>Less people need to be born</h2>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh06vOKbUS6XT-jux6JAmdciGLRqT9ZqP9p8nUGy7giwsvD6cqrmMa7bvQVSN-hjmnO7lzp5OwYl9ac89V-eEkv_403NlnljVJ_-E4vUwUjdh0O8bi6AvQO40b9FXCGoKctrOkhRs1_AlVlAEcoj3Fyns1hbxLCFuQ_DpFjoAZwjhoHAwQLhQ/s1920/overpopulation.014.jpeg" style="display: block; padding: 1em 0; text-align: center; clear: left; float: left;"><img alt="" border="0" width="320" data-original-height="1080" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh06vOKbUS6XT-jux6JAmdciGLRqT9ZqP9p8nUGy7giwsvD6cqrmMa7bvQVSN-hjmnO7lzp5OwYl9ac89V-eEkv_403NlnljVJ_-E4vUwUjdh0O8bi6AvQO40b9FXCGoKctrOkhRs1_AlVlAEcoj3Fyns1hbxLCFuQ_DpFjoAZwjhoHAwQLhQ/s320/overpopulation.014.jpeg"/></a></div>
</p>
<p>Less people need to be born. How do we achieve that?
It is not as controversial as killing people or letting them die. A person who isn't conceived doesn't have any human rights.
An uncontroversial idea is contraceptives to everyone who wants them.</p>
<p>Another popular argument is to educate women since educated and working women
apparently have fewer children.</p>
<p>More controversial is legislation that discourages having many children.
China's controversial 1 child policy was a version of this. It's been abandoned and China now encourages 2 and allows 3 children to stem its aging population.
<a href="https://www.scmp.com/news/china/politics/article/3141860/china-confirms-all-citizens-can-now-have-third-child-amid">https://www.scmp.com/news/china/politics/article/3141860/china-confirms-all-citizens-can-now-have-third-child-amid</a></p>
<div class="separator" style="clear: both;"></div>
<p><h2>More people need to die</h2>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiOZ5AMOyIHgnsrr0TaAV4DyPMLLMnxEfSSBd-o0holMUQnB67Kt5jGoW0rNzv7kGweQEsWkZqNNwTVLxhHErTMytRUBEJFNcuZIxezLTPw2Wy5bo8R1Tyl6UZaF3Q9vFIiVXM7tsbEXHHDDUMZLNoBpvtggxzibFMw4ynXHoUha8QNsxuhow/s1920/overpopulation.013.jpeg" style="display: block; padding: 1em 0; text-align: center; clear: right; float: right;"><img alt="" border="0" width="320" data-original-height="1080" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiOZ5AMOyIHgnsrr0TaAV4DyPMLLMnxEfSSBd-o0holMUQnB67Kt5jGoW0rNzv7kGweQEsWkZqNNwTVLxhHErTMytRUBEJFNcuZIxezLTPw2Wy5bo8R1Tyl6UZaF3Q9vFIiVXM7tsbEXHHDDUMZLNoBpvtggxzibFMw4ynXHoUha8QNsxuhow/s320/overpopulation.013.jpeg"/></a></div></p>
<p>More people need to die. Luckily, we are all mortal and we will eventually all
die. But, is there any way to speed it up without violating human rights?
Abortion? Euthanasia? What else?</p>
<div class="separator" style="clear: both;"></div>
<p>
<h2>The world counts</h2>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiNe6UyEa8ub6-BAippCs0FVrnABYOoq2viJ-nvPcWD10nYGIGqfVJAM1DV5wAr-mWGdzvKC6rC-pOyKtrYeMOe5NySkOBUfSCNaS01fbq1eMzmkkv5ZBsRxLXKBN_LCMBmitYqaQUgnYHASiV9HDfYaw_9RYLEzFUVhaB2o0raWgt9IsglrA/s1920/overpopulation.015.jpeg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="1080" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiNe6UyEa8ub6-BAippCs0FVrnABYOoq2viJ-nvPcWD10nYGIGqfVJAM1DV5wAr-mWGdzvKC6rC-pOyKtrYeMOe5NySkOBUfSCNaS01fbq1eMzmkkv5ZBsRxLXKBN_LCMBmitYqaQUgnYHASiV9HDfYaw_9RYLEzFUVhaB2o0raWgt9IsglrA/s400/overpopulation.015.jpeg"/></a></div>
</p>
<h2>Books</h2>
<ul>
<li><a href="https://www.amazon.com/Under-Sky-We-Make-Warming/dp/0593328175?tag=thtasta-20">Under the sky we make</a></li>
<li><a href="https://www.amazon.com/Billion-Counting-Death-Migration-Shape/dp/B09HYXSVHZ?tag=thtasta-20">8 billions and counting</a></li>
<li><a href="https://www.amazon.com/How-World-Really-Works-Science/dp/0593297067?tag=thtasta-20">How the World Really Works: The Science Behind How We Got Here and Where We're Going</a></li>
</ul>
Anders Janmyrhttp://www.blogger.com/profile/16081537018884349622noreply@blogger.com0tag:blogger.com,1999:blog-34049130.post-30555081260199623512021-04-29T17:09:00.006+02:002021-04-30T10:59:17.145+02:00Global Warming: Time for the Nuclear Option<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5CcKa9DaEUsi5mzOMcGp9dTIX_If9R0Zn7wY_erduT-6GeoWbdPILPFwvz4WVL425c2kLmSGSJYoYAZ48lg_C1H4Ri3KYoVx_g6IE6csv20Ipqjcv1vZ8YliAGCzyyCkeBYo8/s1920/global-warming.001.jpeg" style="display: block; padding: 1em 0; text-align: center; clear: right; float: right;"><img alt="" border="0" width="320" data-original-height="1080" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5CcKa9DaEUsi5mzOMcGp9dTIX_If9R0Zn7wY_erduT-6GeoWbdPILPFwvz4WVL425c2kLmSGSJYoYAZ48lg_C1H4Ri3KYoVx_g6IE6csv20Ipqjcv1vZ8YliAGCzyyCkeBYo8/s320/global-warming.001.jpeg"/></a></div>
<p>The last few months I have spent educating myself about global warming.
I saw the documentary about Greta Thunberg and I figured that she has a point,
so I decided to "Listen to the scientists".</p>
<p>I've read multiple books, finished a couple of Coursera courses and watched
more YouTube videos than a teenager. The conclusion I have come to from all of
this is that it's time to start investing heavily in nuclear power!</p>
<p>The one thing that we can be sure of the coming years is that we will need
a lot of electrical energy. All the energy that is currently used for making
steel and driving cars will need to come from electricity since only
electricity can be created without producing green house gases.</p>
<p>Wind and solar power are great, but they cannot provide the amounts of reliable energy
that will be required.</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg89EqtzoVifLQ76ZKPecs-Npn_HpxXBFTFHIN3GZ2sUhNQs-d6ciKlv8DSYDmDCioqE_r-TZ0suU0l1qBryD9e04PzTjoMEuyj0wpAPDloYOn4lUT8YxcCpLmYLvRPkc_4xWzH/s1920/global-warming.002.jpeg" style="display: block; padding: 1em 0; text-align: center; clear: left; float: left;"><img alt="" border="0" width="320" data-original-height="1080" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg89EqtzoVifLQ76ZKPecs-Npn_HpxXBFTFHIN3GZ2sUhNQs-d6ciKlv8DSYDmDCioqE_r-TZ0suU0l1qBryD9e04PzTjoMEuyj0wpAPDloYOn4lUT8YxcCpLmYLvRPkc_4xWzH/s320/global-warming.002.jpeg"/></a></div>
<p>
Sweden is a tiny country (#55 in the world by area). Our energy usage clocks in
at around 40th place (#25 per capita). You might think that we are an important
country that can make a difference on the climate.</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjc4wSyuUFJXW6d6HhpvP_8loGT6gkstWBeCD7b3BBk9Zb7Iha0Hw_tDDq2Pb1w9yLLdAqqSlk6cGx7bGCMbHpFsgehf28wuKjv5LonG1MIiSOYSrZGJADJWVCzyaGVVNenbsZ2/s1920/global-warming.003.jpeg" style="display: block; padding: 1em 0; text-align: center; clear: right; float: right;"><img alt="" border="0" width="320" data-original-height="1080" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjc4wSyuUFJXW6d6HhpvP_8loGT6gkstWBeCD7b3BBk9Zb7Iha0Hw_tDDq2Pb1w9yLLdAqqSlk6cGx7bGCMbHpFsgehf28wuKjv5LonG1MIiSOYSrZGJADJWVCzyaGVVNenbsZ2/s320/global-warming.003.jpeg"/></a></div>
<p>
WRONG! Sweden is an insignificant statistical blip. We use less than half a
percent of the global energy. China uses about 25 percent and the US around 16
percent.</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgjAdd-hE_NVzIZRq-17BOYPwywKynC_eYy0nPr2YuO_p0J4qUknseVsd4vpzQXrWmNplsGlojB9gnXB3Q4r5zRkeXfFbqn1ziPLZWreiH5kcjMSjZFANbLvZRYDc8ZxyVK818J/s1920/global-warming.004.jpeg" style="display: block; padding: 1em 0; text-align: center; clear: left; float: left;"><img alt="" border="0" width="320" data-original-height="1080" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgjAdd-hE_NVzIZRq-17BOYPwywKynC_eYy0nPr2YuO_p0J4qUknseVsd4vpzQXrWmNplsGlojB9gnXB3Q4r5zRkeXfFbqn1ziPLZWreiH5kcjMSjZFANbLvZRYDc8ZxyVK818J/s320/global-warming.004.jpeg"/></a></div>
<p>
China is currently building one new coal power plant per week in China and
also helping other developing countries build coal plants in exchange for
political power.</p>
<p>How can this be? Isn't China part of the Paris agreement? Yes, but China is
treated as a developing country and, in the name of fairness, doesn't have to
limit it's carbon footprint. The climate doesn't care about fairness.</p>
<p>A Coal plant has an average lifetime of 40 years.</p>
<p>China is by no means the only country trying to expand their fossil fuel usage. It's all over the world!</p>
<p>This means that our hope of reaching the Paris agreement of keeping the
temperature below 2 degrees C is inexistent.</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJaVYQkTWKPAf26h9zi4nUfvw7w8bfY6MdvCzWUABRAceZDX9lvjtawG1WdYb_2QFw6snroweUG53vBZ82N0xcZaBSzvyjw9zWwYXUoIuvdfBJgtfxZPYiFz5353RXZ-NRD0vq/s1920/global-warming.005.jpeg" style="display: block; padding: 1em 0; text-align: center; clear: right; float: right;"><img alt="" border="0" width="320" data-original-height="1080" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJaVYQkTWKPAf26h9zi4nUfvw7w8bfY6MdvCzWUABRAceZDX9lvjtawG1WdYb_2QFw6snroweUG53vBZ82N0xcZaBSzvyjw9zWwYXUoIuvdfBJgtfxZPYiFz5353RXZ-NRD0vq/s320/global-warming.005.jpeg"/></a></div>
<p>
So, what should we do? Should we just shrug and say "Fuck it, we're screwed
anyway".</p> <p>I don't think so. Sweden is a technologically advanced country and we
can be an example. We can show the world what can be done.</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhqjoa78bmGaxXmd1flpc9yagybmgz-rzA_iwtAn7YFxSpEKSlbr9YoZlzEbwHoZg-f8PG4-kC6Bpco_suPWrz98qkK9EAw5cJDXaRE4I4pfn1IgsWcRL1ZW3Z828l3Yusxj-sa/s1920/global-warming.006.jpeg" style="display: block; padding: 1em 0; text-align: center; clear: left; float: left;"><img alt="" border="0" width="320" data-original-height="1080" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhqjoa78bmGaxXmd1flpc9yagybmgz-rzA_iwtAn7YFxSpEKSlbr9YoZlzEbwHoZg-f8PG4-kC6Bpco_suPWrz98qkK9EAw5cJDXaRE4I4pfn1IgsWcRL1ZW3Z828l3Yusxj-sa/s320/global-warming.006.jpeg"/></a></div>
<p>
What has Sweden done so far? A tax on plastic bags. For fuck sake!</p>
<p>There has also been other suggestions, but all of them are about the future. In
Sweden, just as in every other country, politicians love to make bold promises
about the future, instead of acting in the present. Promises, doesn't cost
anything, but they don't improve anything either.</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgn2Z4yWgkV7OJugw0DKz-Z81CI6F72ZlkFNr-cyYuWDK8zCe_HBGxXz0QllxaMe8W_VJKxs0Bf4-dSi5Rt1_C3bZQCWWY4OoXQOo21Mf4NWIvF_JVUl6-Y-cuVoHKosoIZHX1d/s1920/global-warming.007.jpeg" style="display: block; padding: 1em 0; text-align: center; clear: right; float: right;"><img alt="" border="0" width="320" data-original-height="1080" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgn2Z4yWgkV7OJugw0DKz-Z81CI6F72ZlkFNr-cyYuWDK8zCe_HBGxXz0QllxaMe8W_VJKxs0Bf4-dSi5Rt1_C3bZQCWWY4OoXQOo21Mf4NWIvF_JVUl6-Y-cuVoHKosoIZHX1d/s320/global-warming.007.jpeg"/></a></div>
<p>
Almost 40% of Sweden's energy consumption is fossil-based. All of it has to become fossil-free.</p>
<p>This requires a lot of fossil-free energy.</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEis6TH76_jFDJ_L-U-Wzp_XmW2ELtsaOYGLnVj0sbVurg-1tb-lsYhbvAkkDjdKTBQWG5MUZU0F9frMmUFO3XzTwYr5iP_p29HTr1vrhmugXdelw2OH44MGjxSw0GA4ERDDK_Iq/s1920/global-warming.008.jpeg" style="display: block; padding: 1em 0; text-align: center; clear: right; float: right;"><img alt="" border="0" width="320" data-original-height="1080" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEis6TH76_jFDJ_L-U-Wzp_XmW2ELtsaOYGLnVj0sbVurg-1tb-lsYhbvAkkDjdKTBQWG5MUZU0F9frMmUFO3XzTwYr5iP_p29HTr1vrhmugXdelw2OH44MGjxSw0GA4ERDDK_Iq/s320/global-warming.008.jpeg"/></a></div>
<p>
HYBRIT, fossil-free steel is a really cool invention that is currently worked
on by Swedish mining companies. The coal normally used in the creation of steel
will be replaced by hydrogen produced by water and fossil-free energy.</p>
<p>This requires a lot of fossil-free energy.</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi9aeGrEanPzHmUvtOwVI286ce19Yoir1A0p1vnS5WYy2k7YSsgJLiXmD4Ooh6GbXadNb9N-5lbchWrPp9mthiPr9ThM7X7TwmNIA9RBVHMb5cYyripdPxV5a4Lc5vu1AtBRkGo/s1920/global-warming.009.jpeg" style="display: block; padding: 1em 0; text-align: center; clear: left; float: left;"><img alt="" border="0" width="320" data-original-height="1080" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi9aeGrEanPzHmUvtOwVI286ce19Yoir1A0p1vnS5WYy2k7YSsgJLiXmD4Ooh6GbXadNb9N-5lbchWrPp9mthiPr9ThM7X7TwmNIA9RBVHMb5cYyripdPxV5a4Lc5vu1AtBRkGo/s320/global-warming.009.jpeg"/></a></div>
<p>
Wind and solar produces less than 10% of the energy in Sweden.
Currently, nuclear produces about 27%, down from 33%. This means that almost
all of the carbon reductions we have achieved from building out solar and wind
has been lost due to nuclear reductions.</p>
<p>If we instead of reducing nuclear had increased it, Sweden could have been
almost carbon neutral now!</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9cz83W1tg0HWdrdLORy-Bjyge4XvEI13fRukv4fJN4b1gnRtie5szvI5M-oHGhuayXG0jGIA_itV6DcZ2cKCsUjg_1YHW-xOP5HikdNtLM57stuMkB5PbPNKF107TmlNT-vWC/s1920/global-warming.010.jpeg" style="display: block; padding: 1em 0; text-align: center; clear: right; float: right;"><img alt="" border="0" width="320" data-original-height="1080" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9cz83W1tg0HWdrdLORy-Bjyge4XvEI13fRukv4fJN4b1gnRtie5szvI5M-oHGhuayXG0jGIA_itV6DcZ2cKCsUjg_1YHW-xOP5HikdNtLM57stuMkB5PbPNKF107TmlNT-vWC/s320/global-warming.010.jpeg"/></a></div>
<p>
Nuclear has the smallest carbon footprint of any energy source. It's even lower
that Wind and Solar.</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgozbDpZnssTVSzCZ_tDtrM_px3wsT5_DZjODbYWEcy9BIBCx-og7aMhmVEF36ETQKzUqp-7jOCYvRPz7wJ5kFn0yTCszWlrTfXqmO5gQ_ryrGeoIj1za7ayByWAQ-WEgBAZN9s/s1920/global-warming.011.jpeg" style="display: block; padding: 1em 0; text-align: center; clear: left; float: left;"><img alt="" border="0" width="320" data-original-height="1080" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgozbDpZnssTVSzCZ_tDtrM_px3wsT5_DZjODbYWEcy9BIBCx-og7aMhmVEF36ETQKzUqp-7jOCYvRPz7wJ5kFn0yTCszWlrTfXqmO5gQ_ryrGeoIj1za7ayByWAQ-WEgBAZN9s/s320/global-warming.011.jpeg"/></a></div>
<p>
But, what about the risk of accidents? The whole world will be contaminated if
we have a nuclear meltdown. The facts tell a different story.</p>
<p>Chernobyl, the worst accident by far, has killed less than 60 people, to date.
Fukushima has produced 1 dead due to cancer caused by radiation (most people
died in the Tsunami).
Three Mile Island, the accident that caused the nuclear scare in the 70s,
produced zero deaths.</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjFFHnIkI8xl5jB_khEMp2PjnZIvvmjDiWVKbG3Hofqyjhgk-vYv86S-YgKO0t7zeRteZhVoR9-SXO_pXtPoMqPsWssiF_CFhCoz_BzAMwlnMbU00gRYPnyURhvEZqHLiVNXRqt/s1920/global-warming.012.jpeg" style="display: block; padding: 1em 0; text-align: center; clear: right; float: right;"><img alt="" border="0" width="320" data-original-height="1080" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjFFHnIkI8xl5jB_khEMp2PjnZIvvmjDiWVKbG3Hofqyjhgk-vYv86S-YgKO0t7zeRteZhVoR9-SXO_pXtPoMqPsWssiF_CFhCoz_BzAMwlnMbU00gRYPnyURhvEZqHLiVNXRqt/s320/global-warming.012.jpeg"/></a></div>
<p>
Nuclear power has killed less than a 100 people in 50 years.</p>
<p>Carbon pollution is estimated to be killing more than three million people per
year.</p>
<p>Update: According to some models, link below, Chernobyl is expected to kill +4000 people from long term low-level radiation exposure. It doesn't invalidate my point.</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEif6tdskPRTdej7TXnTsOVzrFAOZ2XlXxWaF-wum0iXDNq9LQvfDB7VZeEAAXdf0Kkb402QdYCOyvYiO5oQGpzCt1KTXJALc1cPr3cGgATQN_nTkyp2IjVvkQzpZOw_cjEg4xx0/s1920/global-warming.013.jpeg" style="display: block; padding: 1em 0; text-align: center; clear: left; float: left;"><img alt="" border="0" width="320" data-original-height="1080" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEif6tdskPRTdej7TXnTsOVzrFAOZ2XlXxWaF-wum0iXDNq9LQvfDB7VZeEAAXdf0Kkb402QdYCOyvYiO5oQGpzCt1KTXJALc1cPr3cGgATQN_nTkyp2IjVvkQzpZOw_cjEg4xx0/s320/global-warming.013.jpeg"/></a></div>
<p>
When accidents have been shown to be insignificant, the elephant in the nuclear
room is nuclear waste.</p>
<p>Nuclear waste is classified into three categories, High, Intermediate and Low
Level Waste. Small parts of the High Level Waste (HLW) can stay active for 1000
years.</p>
<p>Only, 3 percent of waste is classified as HLW. If the waste is reprocessed,
as it is in France, the amount is reduced to 0.2 percent.</p>
<p>According to the Nuclear Industry:</p>
<blockquote>They have developed – and implemented – most of the necessary technologies required for the final disposal of all of the waste it produces. The remaining issue is one of public acceptance, and not of technological feasibility.
</blockquote>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgXuFttCJAViFPCXvWpJhvM3Yt4m1hwYgrEJaMwY7-s2YUIb9iIv8Zn3zTttAe8Fzy6jEXf-NKp8Djz3-2HF_iPq1cDp8VhmBJNMvFJaKWkkkQ5wzFm9UGL1zA98IU0Kh4YI36K/s1920/global-warming.014.jpeg" style="display: block; padding: 1em 0; text-align: center; clear: left; float: left;"><img alt="" border="0" width="320" data-original-height="1080" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgXuFttCJAViFPCXvWpJhvM3Yt4m1hwYgrEJaMwY7-s2YUIb9iIv8Zn3zTttAe8Fzy6jEXf-NKp8Djz3-2HF_iPq1cDp8VhmBJNMvFJaKWkkkQ5wzFm9UGL1zA98IU0Kh4YI36K/s320/global-warming.014.jpeg"/></a></div>
<p>I don't know enough about nuclear waste storage to know if this is the case or
not, but even if the waste problem is not solved, we need to move forward with
Nuclear.</p>
<p>The experts agree that global warming based on our carbon emissions is putting future generations at risk.</p>
<p>The responsible thing to do it to expand the most effective way to create carbon-free energy.</p>
<p> The time is ripe for nuclear power.</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiI-AW-i1aB4r_hk96p7qSnWd7IUJZdfO5mJVacE3p3SnrMss9vE-2C5UAqsROIs4RCjR1B_Rj-WuvHf4aod3YaAMnSAjQ7s7VRVRmZXRyZMucpft8yxNNCSS9A12BlpjwUnfjj/s1920/global-warming.015.jpeg" style="display: block; padding: 1em 0; text-align: center; clear: right; float: right;"><img alt="" border="0" width="320" data-original-height="1080" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiI-AW-i1aB4r_hk96p7qSnWd7IUJZdfO5mJVacE3p3SnrMss9vE-2C5UAqsROIs4RCjR1B_Rj-WuvHf4aod3YaAMnSAjQ7s7VRVRmZXRyZMucpft8yxNNCSS9A12BlpjwUnfjj/s320/global-warming.015.jpeg"/></a></div>
<p>Next time you hear someone say that we need to do this or that for the environment,
ask "How?"</p>
<p> "How, will this make a significant impact?".</p>
<p> "How, will this make us carbon neutral?".</p>
<p>If they can't answer that, they're just wasting time.</p>
<div class="separator" style="clear: both;">
<h1 id="references">References</h1>
<h3 id="books">Books</h3>
<p><a href="https://www.amazon.com/How-Avoid-Climate-Disaster-Breakthroughs/dp/0385546130/?tag=thtasta-20">How to avoid a climate disaster by Bill Gates</a>
This is a great book that has been criticised since Gates is a high poluter that likes to eat hamburgers instead of by the information that it contains. He clearly highlights all the challenges we are facing and does it through an economic perspective that targets business people. If you are not ecomically inclined you may see this as a drawback, but since the move to carbon-neutral must be financed I think that it is a great way to frame it.
</p>
<p><a href="https://www.amazon.com/Apocalypse-Never-Environmental-Alarmism-Hurts/dp/0063001691/tag=thtasta-20">Apocalypse Never by Michael Shellenberger</a> Michael Shellenberger is an environmentalist that has become fed up with alarmist communication and tries to tackle many environmental problems with facts. A fantastic book! Among other things, he puts the upcoming catastrophe into perspective. It's not the end of the world.</p>
<p><a href="https://www.adlibris.com/se/bok/klimatkatastrofens-psykologi-och-mansklighetens-framtid-det-etiska-alternativet-9789163978821">Klimatkatastofens psykologi by Billy Larsson</a>: Billy Larsson is a psychologist who was written a comprehensive evaluation of many the problems we are facing with climate change. The author doesn't have high hopes that we will pull through this and has started to invest in a farm to do his part to live ecologically.
</p>
<p><a href="https://www.amazon.com/This-Changes-Everything-Capitalism-Climate/dp/1451697392/?tag=thtasta-20">This Changes Everything by Naomi Klein</a> Noami Klein believes that global warming will force the world out of "deregulated capitalism". Even though I agree with many of the things she writes about, I don't have any hopes that this will force this change. Nevertheless, it's a good book worth reading.</p>
<p><a href="https://www.amazon.com/Six-Degrees-Future-Hotter-Planet/dp/1426203853/?tag=thtasta-20">Six Degrees by Mark Lynas</a> Mark Lynas describes the effetct of Global Warming, known as "Global Weirding" as the temperature rises. It's an interesting book showing many of the worst case scenarios that may occur in the future.</p>
<p><a href="https://www.wiley.com/en-se/Global+Warming%3A+Understanding+the+Forecast%2C+2nd+Edition-p-9781118213971">Global Warming: Understanding the Forecast by David Archier</a> This book is the litterature behind the Coursera Global Warming course, linked below. It's a in depth book about how the systems of the earth works and how the scientists can know that we are warming up the globe.</p>
<p><a href="https://www.amazon.com/Global-Warming-Very-Peculiar-History/dp/1907184511/?tag=thtasta-20">Global Warming: A Very Peculiar History</a> A concise history of global warming. </p>
<p><a href="https://www.adlibris.com/se/bok/atta-steg-mot-avgrunden-vart-framtida-liv-pa-planeten-9789177750871">Åtta steg mot avgrunden</a> Another book about global weirding. One chapter per decade until the end of the century.</p>
<p><a href="https://www.amazon.com/Drive-Nuclear-Reactor-Springer-Praxis/dp/3030338754/?tag=thtasta-20">How to Drive a Nuclear Reactor by Colin Tucker</a> A tour of the science behind nuclear reactors, through their start-up, operation and shutdown.</p>
<h3 id="coursera">Coursera</h3>
<p><a href="https://www.coursera.org/learn/global-warming/home/welcome">Global Warming I: The Science and Modeling of Climate Change</a></p>
<p><a href="https://www.coursera.org/learn/global-warming-model">Global Warming II: Create Your Own Models in Python</a></p>
<h3 id="videos">Videos</h3>
<p><a href="https://www.youtube.com/watch?v=N-yALPEpV4w">Why renewables can’t save the planet | Michael Shellenberger</a></p>
<p><a href="https://www.youtube.com/watch?v=bNKdlnoAqIs">Bill Gates: The 2021 60 Minutes Interview</a></p>
<p><a href="https://www.youtube.com/watch?v=PODbE4v80Yg">The Climate for Nuclear | Daniel Poneman</a></p>
<p><a href="https://www.youtube.com/watch?v=ciStnd9Y2ak">Why I changed my mind about nuclear power | Michael Shellenberger</a></p>
<h3 id="articles">Articles</h3>
<p><a href="https://ourworldindata.org/energy/country/sweden?country=~SWE">Sweden: Energy Country Profile</a></p>
<p><a href="https://www.greenamerica.org/fight-dirty-energy/amazon-build-cleaner-cloud/10-reasons-oppose-nuclear-energy">Ten Reasons to Oppose Nuclear Energy</a></p>
<p><a href="https://www.world-nuclear.org/information-library/nuclear-fuel-cycle/nuclear-wastes/radioactive-wastes-myths-and-realities.aspx">Radioactive Waste – Myths and Realities</a></p>
<p><a href="https://ourworldindata.org/what-was-the-death-toll-from-chernobyl-and-fukushima">What was the death toll from Chernobyl and Fukushima?</a></p>
Anders Janmyrhttp://www.blogger.com/profile/16081537018884349622noreply@blogger.com2tag:blogger.com,1999:blog-34049130.post-53988468049062020102020-11-17T17:44:00.000+01:002020-11-17T17:44:40.987+01:00No Smoking Policy<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgDryqRfl7CAqfHRUa929XfvQSsLdAZ8ZLUgFU68ErSS884ERZKS7zlTY87vPZ23FKgW580OE2RFd1HOOpeZDRDIn4fxokmS1JWomljao5w6ur-v5dObmkK8SgJYlvddKH_EQZT/s0/no-smoking.jpg" style="display: block; padding: 1em 0; text-align: center; clear: left; float: left;"><img alt="" border="0" data-original-height="600" data-original-width="800" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgDryqRfl7CAqfHRUa929XfvQSsLdAZ8ZLUgFU68ErSS884ERZKS7zlTY87vPZ23FKgW580OE2RFd1HOOpeZDRDIn4fxokmS1JWomljao5w6ur-v5dObmkK8SgJYlvddKH_EQZT/s0/no-smoking.jpg"/></a></div>Anders Janmyrhttp://www.blogger.com/profile/16081537018884349622noreply@blogger.com0tag:blogger.com,1999:blog-34049130.post-22163206979783093692020-11-14T12:12:00.000+01:002020-11-14T12:12:13.472+01:00In the Present<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhskFVEl2j757G-4_U0WFelfMBv8HtUAWznsWm3BFnfDPGyCiiVXwycaFeVh_wO19TYhhe8Gi6A1ChBz45eBKINsbLLqYyR96YfG_QLCo9CFID9XW45sn1O9bLmSmH4oZhUIMVK/s0/in-the-present.png" style="display: block; padding: 1em 0; text-align: center; clear: left; float: left;"><img alt="" border="0" data-original-height="600" data-original-width="600" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhskFVEl2j757G-4_U0WFelfMBv8HtUAWznsWm3BFnfDPGyCiiVXwycaFeVh_wO19TYhhe8Gi6A1ChBz45eBKINsbLLqYyR96YfG_QLCo9CFID9XW45sn1O9bLmSmH4oZhUIMVK/s0/in-the-present.png"/></a></div>Anders Janmyrhttp://www.blogger.com/profile/16081537018884349622noreply@blogger.com0tag:blogger.com,1999:blog-34049130.post-72716831146671444562020-10-29T17:44:00.000+01:002020-10-29T17:44:02.665+01:00The Corona Virus from a Tapir's Perspective<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhgUYEADtgaPfL3rCva9l2G9b8aWz5HN21JUZ8mEilhVHNsGBR1LTQLLcGdyVsswBFXOttyRpDYud4KB62GU2lRIlmi9P7VaWi6Dn25k3VcSW40d6hyphenhyphenoVCLREI_pCdrRWnPuA35/s240/tapir-gazette.png" style="display: block; padding: 1em 0; text-align: center; clear: right; float: right;"><img alt="" border="0" width="320" data-original-height="160" data-original-width="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhgUYEADtgaPfL3rCva9l2G9b8aWz5HN21JUZ8mEilhVHNsGBR1LTQLLcGdyVsswBFXOttyRpDYud4KB62GU2lRIlmi9P7VaWi6Dn25k3VcSW40d6hyphenhyphenoVCLREI_pCdrRWnPuA35/s320/tapir-gazette.png"/></a></div>
<p><strong>Tapir Gazette: So, what do you think about the latest conflict between
the Humans and the Corona Viruses?</strong></p>
<p><strong>Tony:</strong> It's about fucking time someone fought back against those bastards.
They've been bullying all other species for far too long!</p>
<p><strong>Tara:</strong> Hear, hear! I've been hoping someone would take them on for years,
but there seemed to be no stopping them.</p>
<p><strong>Tuck:</strong> Yes, they have been eradicating the rest of us at their expense for
hundreds of years now. I'm glad someone is sticking it to them. Go Corona!</p>
<p><strong>Tara:</strong> The worst part is that they think they have a right to this since
they are they Kings of the world. Who wears the crown (corona) now, motherfuckers!</p>
<p><strong>Tapir Gazette: What about the strategy of the Corona Viruses?</strong></p>
<p><strong>Tanya:</strong> It was looking pretty good in the beginning, when the mortality rate
was about 6%. That could really have made a dent in the population, perhaps
enough to keep them indoors. But, now it looks like it's more 0.6% and that
won't make a difference in the long run.</p>
<p><strong>Tuck:</strong> I think the Coronas should have been more patient. They should have
waited longer before they started taking people out and just infected as many
as they could before striking.</p>
<p><strong>Tanya:</strong> Yeah, I agree, they jumped the gun. It would have been better to
stay under the radar for a longer time before coming out in force.</p>
<p><strong>Tara:</strong> But it was a smart move to strike in the dictatorships first. It
gave them more time to spread to the rest of the world since the dictatorships care more
about appearances than about people.</p>
<p><strong>Tapir Gazette: What about the human tactics?</strong></p>
<p><strong>Tony:</strong> From my perspective the "Shelter in Place" tactics is awesome! The
more we keep the humans indoors the better it is for everyone else. But, from
a human perspective, I don't know. I don't think it will last very long if the
mortality rate stays below 1%.</p>
<p><strong>Tuck:</strong> Humans have become so complacent, they can't handle a crisis like
this. If they can't find toilet paper at the store, they riot!</p>
<p><strong>Tapir Gazette: Why do you dislike the humans so much?</strong></p>
<p><strong>Tuck:</strong> What's not to dislike? They are the most <em>speciest</em> (racists for species)
assholes on earth. They care about nothing but themselves. Even the richest, so called
philanthropists, only care about helping <i>people</i> at the expense of the rest of
us.</p>
<p><strong>Tanya:</strong> Yeah, have you seen the population curves? They are totally crazy.</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhBSw9AZY_2-UeI_UAHbpkpCSSl1LlPxIdVAE1ZJ_cUTqkGXTecgEySLxmSEpsWC0tnbiB7vjacgndNBYCsFQMEZ93g_xZo3_OGcIqvKQBhVvpyVhtdlKAaScFKUx46uB8G-S5G/s2048/world-population-by-world-regions-post-1820.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="320" data-original-height="1446" data-original-width="2048" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhBSw9AZY_2-UeI_UAHbpkpCSSl1LlPxIdVAE1ZJ_cUTqkGXTecgEySLxmSEpsWC0tnbiB7vjacgndNBYCsFQMEZ93g_xZo3_OGcIqvKQBhVvpyVhtdlKAaScFKUx46uB8G-S5G/s320/world-population-by-world-regions-post-1820.png"/></a></div>
<p><strong>Tony:</strong> Yeah, they are the real pandemic!</p>
<p><strong>Tapir Gazette: Any final words, anyone?</strong></p>
<p><strong>Tuck:</strong> I hope it's not my final words, but who knows these days ;). I don't
think that all humans are assholes, some of them try to make a difference. The
latest report from <a href="https://livingplanet.panda.org/en-us/">Living Planet</a> shows
that at least some people seem to care.</p>
<p><strong>Tanya</strong>: I wish I had your positive mindset Tuck, but I can't see anything
bright about our future. The only good human is a dead human. At least they are good fertilizers!</p>
<p><strong>Note</strong>: <em>The names of the tapirs have been changed to protect them from retaliation.</em></p>
Anders Janmyrhttp://www.blogger.com/profile/16081537018884349622noreply@blogger.com0tag:blogger.com,1999:blog-34049130.post-82754911624844559882020-01-04T13:16:00.000+01:002020-01-04T13:16:38.987+01:00Three Years in Silicon Valley
<p>Three years ago, I moved with my family, wife, two kids and two dogs, from Sweden to California to work for a
subsidiary of a Swedish company called Jayway Inc.</p>
<p>Jayway Inc. was located in Palo Alto, the heart of Silicon Valley. Silicon
Valley is a wide valley that is situated between two mountain ranges, the Santa
Cruz Mountains along the cost, and the Diablo Range on the other side of the
bay. The area is known as "the Bay Area" to locals since the bay stretches
through the whole valley from San Francisco in the north to San José in the
south.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhNJGgs7Ri-r6wRz0Zwg1eL4DeCRb1y7I67YLEX0a0BJ1hl8URJKV_i5Ay684UTWAkm6Fna6ppkkwcy0xemBh_a-LRS2rHP521YqQ5eiV92XfmH96LL_HmTWaKPSUMvzWwkxIrP/s1600/bayarea.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhNJGgs7Ri-r6wRz0Zwg1eL4DeCRb1y7I67YLEX0a0BJ1hl8URJKV_i5Ay684UTWAkm6Fna6ppkkwcy0xemBh_a-LRS2rHP521YqQ5eiV92XfmH96LL_HmTWaKPSUMvzWwkxIrP/s320/bayarea.jpg" width="320" height="274" data-original-width="598" data-original-height="512" /></a></div>
<p>San José, the lesser know of the two cities is actually the larger
of the two with a population of more than 1 million. San Francisco has around
800 thousand.</p>
<h1>Working</h1>
<p>During my three years I mainly worked for Walmart and it's membership part,
Sam's Club. Walmart is the largest private company in the world, both in terms
of revenue, $500 billion, and in terms of employees, 2.2 million. There are two
non-private organizations which have more employees than Walmart, the U.S. and
the Chinese armies.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3lswLxslW6bxNFPrnE0FJ7YzY0rFc7hPFy8-LcA70tFlcHEo4CmEWNpw4Oj4lMzdKaSiwyyRGX8cG7VBQXKOaGji_AJSiRhXJmffjh0D-n6eAkS5ZVeDN7d9Jmg_w3BVB7RrI/s1600/walmart.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3lswLxslW6bxNFPrnE0FJ7YzY0rFc7hPFy8-LcA70tFlcHEo4CmEWNpw4Oj4lMzdKaSiwyyRGX8cG7VBQXKOaGji_AJSiRhXJmffjh0D-n6eAkS5ZVeDN7d9Jmg_w3BVB7RrI/s320/walmart.jpg" width="320" height="137" data-original-width="692" data-original-height="296" /></a></div>
<p>Working in a huge organization like this may be intimidating, but the teams I
worked with were surprisingly nimble and we were expected to take initiatives
and responsibility for what we did. There was no micro-management and, if
anything, I have had to little input on what direction to move in.</p>
<p>The three projects I worked in were very disparate.</p>
<p>The first was the test tools team, where I helped out to build tools for
testing mobile applications "in the cloud" with Sauce Labs and Firebase Test
Lab.</p>
<p>My second project was a re-write of the backend for Sam's club's Scan and Go
application. An application where you use your mobile phone to scan products
and checkout without having to stand in line.</p>
<p>In my third project, I worked on Kubernetes for WCNP, the Walmart Cloud Native
Platform. A cross-platform offering that allows Walmart teams to deploy their
applications to GCP, Azure, and one of 5000 VM-Ware installations in the
stores and distribution centers. Quite a challenge.</p>
<h1>Leisure Time</h1>
<p>When not working, it's possible to do anything in the Bay Area.</p>
<p>The mountains surrounding the bay, twenty minutes away, allows for hiking and
mountain biking. On a typical trip you will see wild turkeys, coyotes and deer,
and if you are really lucky, mountain lions.</p>
<p>On the coast, the waves are spectacular and provides among the best surfing in
the world. Half Moon Bay, and Maverick's beach is host to a yearly big wave
surf competition known as Titans of Mavericks.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/YtSFXW3Z3MM?start=31" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhd3Zrn2WnMPQsbm8EFh7ENMISLzSsiQSqeJKO1WwPisHj_X0_fazfMOpWamdDjngvlmVKaYRXXIkujkELmgLYi1EfQDlsgdeMGznEbupe-nfgXIvtlAMarVWPkqGWQdlgdR-qQ/s1600/IMG_2108.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhd3Zrn2WnMPQsbm8EFh7ENMISLzSsiQSqeJKO1WwPisHj_X0_fazfMOpWamdDjngvlmVKaYRXXIkujkELmgLYi1EfQDlsgdeMGznEbupe-nfgXIvtlAMarVWPkqGWQdlgdR-qQ/s320/IMG_2108.jpg" width="320" height="240" data-original-width="1280" data-original-height="960" /></a></div>
<p>If you are fond of climbing, Planet Granite has three climbing centers in the
bay area and if you want to climb outdoors, Yosemite National Park is less than
4 hours away. I recommend watching "Valley Uprising" and "Free Solo" before you
go.</p>
<p>There is also iFly if you want to learn how to skydive in a wind tunnel or a
number of small private airports if you want to take flying lessons.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhhdprhGTutmpE0STtga4zOFzbg2YEFX1fbQ9pUmeQGeecz12GJTyrW7WxKrJIrxqNhyphenhyphenak6fGzLbcbR_ooIoGgkrxGksrIVJV0sPARjJimgS_5rpB97l8I-VygcsCn3SallkAse/s1600/IMG_20180316_091738.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhhdprhGTutmpE0STtga4zOFzbg2YEFX1fbQ9pUmeQGeecz12GJTyrW7WxKrJIrxqNhyphenhyphenak6fGzLbcbR_ooIoGgkrxGksrIVJV0sPARjJimgS_5rpB97l8I-VygcsCn3SallkAse/s320/IMG_20180316_091738.jpg" width="320" height="240" data-original-width="653" data-original-height="490" /></a></div>
<p>For skiing, Lake Tahoe is 4 hours away, depending on traffic. Our longest trip
took 12 hours.</p>
<p>Lake Tahoe holds a number of skiing resorts. During our time in the states we tried out Heavenly, Kirkwood, North Star and Squaw Valley. They were all good for different reasons.</p>
<br style="clear:both;">
<h1>Traffic</h1>
<p>Traffic is awful! The 101, a 5 lane highway through the heart of the
valley is congested during rush hours. Rush hours are between 7 and 10 in the
morning and 4 through 7 in the afternoon. At this time it will take you at
least twice as long as it normally does to go from one place to another. This
means it will take two hours to go from San José to San Francisco, if there are
no accidents!</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjGIFW9pqpEYZI8iHRZQFNlRGNVQYdzd5wsS0Y92fdiLXxNhybFimrmm0xquAlMdYkNqzzddTCFKBy3I0P-GJZQKEvWwAauaKioXV3HsvB8tOXnMYbJHkgtsVNCngcI7UXJKlOq/s1600/101.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjGIFW9pqpEYZI8iHRZQFNlRGNVQYdzd5wsS0Y92fdiLXxNhybFimrmm0xquAlMdYkNqzzddTCFKBy3I0P-GJZQKEvWwAauaKioXV3HsvB8tOXnMYbJHkgtsVNCngcI7UXJKlOq/s320/101.jpg" width="320" height="119" data-original-width="833" data-original-height="309" /></a></div>
<p>The 101 is also far from a safe highway. Many places have potholes the size of
a tire and if you are not paying attention you may easily get a flat.</p>
<p>Taking trips to Tahoe or Yosemite on popular times, such as the weekend, will
also take you a long time even if the weather conditions are good. If it's a
snow storm, forgettaboutit!</p>
<p>One way to speed up your travel is to use the car pool lanes. This is one or
two lanes that are dedicated to people with two or more people in it. This
helps, but since you may also use it if you drive an electric car, in the area
where there is most Tesla's per capita, it doesn't help much.</p>
<p>And since this is the U.S. of A., there is another solution, Pay for it. In many
areas, the car pool lane has been replaced with a Fast-Track lane, it allows
you to pay to use the lane and the price depends on what time of day you are
driving. Why should the rich have to wait for poor people? That is just silly!</p>
<p>Americans also drive differently than Swedes. Many drivers choose one of the
lanes and then stick to that lane for the entire trip, no matter what speed
they are going. It's quite a contrast to Sweden where the left lanes are used
for passing and you normally drive in the right-most lane.</p>
<h1>Caltrain</h1>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjEPc4c6VOv2OSWhXQo36ybLuWpsDAE8PxBK9wVyXfDUVaRkKvuMaggMAjyTFU4YGuNFZljobkEBgrbPPr6kOueEhAZJqJfzrjg_Tew1mIDN9qDtuB3X8MqcENWoFhz8q8Wo-3E/s1600/caltrain.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjEPc4c6VOv2OSWhXQo36ybLuWpsDAE8PxBK9wVyXfDUVaRkKvuMaggMAjyTFU4YGuNFZljobkEBgrbPPr6kOueEhAZJqJfzrjg_Tew1mIDN9qDtuB3X8MqcENWoFhz8q8Wo-3E/s320/caltrain.jpg" width="310" height="194" data-original-width="310" data-original-height="194" /></a></div>
<p>Caltrain is a train that runs through the entire valley, from SF in the north
all the way down to Gilroy in the south. It is diesel powered and feels very
dated compared to the Swedish trains. The road crossings are amazingly stupid.
Most are in the same plane as the roads and often have a traffic light on the
other side of the track with room for one car. It is easy to become stuck on
the tracks when the lights switch to red. A lot of accidents has happened
because of this.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhN_EDep0LBXMr8JE6lhfpj8_MQh1ctiq0MPLX77QRpvryhk517DtEHhE21imDf277n9uBk7wsXUOdACFCh_7QI3iUqSSCj483VLB6mI8sE7_Wksuw5vZulxMEJanRDWch4Mab_/s1600/caltrain-bikes.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhN_EDep0LBXMr8JE6lhfpj8_MQh1ctiq0MPLX77QRpvryhk517DtEHhE21imDf277n9uBk7wsXUOdACFCh_7QI3iUqSSCj483VLB6mI8sE7_Wksuw5vZulxMEJanRDWch4Mab_/s320/caltrain-bikes.jpg" width="259" height="194" data-original-width="259" data-original-height="194" /></a></div>
<p>Not everything is bad about the trains though. The cars have two floors and can
fit quite a lot of people. But, the best part is the bicycle cars. These are
cars where the lower floor is for bicycles and they take around 40 per car.
This is awesome and better than anything I have seen anywhere in Europe.</p>
<p>The bikes are stacked along the walls and the loading and packing is managed by
the riders. It is simplified by most bikes having a label describing its
destination.</p>
<h1>Money</h1>
<p>Everything costs money here. A membership in the YMCA, one of the cheaper
training facilities, costs about $80 a month. If you want to go to a more fancy
place, it can easily set you back $250 a month.</p>
<p>Lunch at an okay restaurant will cost you about $15 with water to drink and no
tip included. The normal tip for food and drink is 18 percent. Tipping at a
service facility such as a massage parlor is 20 to 25 percent. Get used to it.
That's how it works here.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjYUOBl7XZR4cLEziLvph0BiCThtTi1xJ_ucGWk6z9NYz9F4SjjYC_-UZwySVTYS_dGk_V6waLlr5SoFORhtQmVacHZpe2LlzgsQ7v7s-WHuAXk8ljGGQnjVPrzOP6VVCn3WedS/s1600/weed.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjYUOBl7XZR4cLEziLvph0BiCThtTi1xJ_ucGWk6z9NYz9F4SjjYC_-UZwySVTYS_dGk_V6waLlr5SoFORhtQmVacHZpe2LlzgsQ7v7s-WHuAXk8ljGGQnjVPrzOP6VVCn3WedS/s320/weed.jpg" width="320" height="180" data-original-width="520" data-original-height="292" /></a></div>
<p>But, if you spend money, you can buy anything. There are wineries and breweries
everywhere, not just in Napa and Sonoma. Want to smoke some weed? Just go to
any of the local dispensaries or have it delivered to your door. Cannabis is
legal in California for both medical and recreational use!</p>
<p>What else can you have delivered? Any food can be delivered by DoorDash,
GrubHub, UberEats or any of tens of other delivery services.</p>
<p>Want to have yourself delivered? Order an Uber or a Lyft and you will have a
car at your door in less than five minutes that takes you where you want to go.
It even works at 5 in the morning when you need to catch an early flight.</p>
<p>Going to the doctor or dentist may also cost you if you are not paying
attention. Even though the Jayway insurance was really good, a doctor may still
(a dentist will definitely) suggest that you get a procedure that is not
included. Always ask "Is it included in my insurance?" before you say yes to
anything.</p>
<h1>Health Care</h1>
<p>The health care system is very good, if you have good insurance. The health
care providers I was in contact with had very short access times and it was
easy to an appointment. They were also proactive and scheduled a variety of
different checkups to help me stay aware of my health.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg4H72q4eX70_IwgYw32RDd65ode56YEVJ04kUG8LoIO9l9Hf3zx4oJZpSakn_uEYB9cCTRgLK0X4S6yUuaPMDrulOo1TfYYtJZMA2RMBvh4gcqoIHYg8z125JbE9-SD9i-Pu96/s1600/PA_campus_building.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg4H72q4eX70_IwgYw32RDd65ode56YEVJ04kUG8LoIO9l9Hf3zx4oJZpSakn_uEYB9cCTRgLK0X4S6yUuaPMDrulOo1TfYYtJZMA2RMBvh4gcqoIHYg8z125JbE9-SD9i-Pu96/s320/PA_campus_building.jpg" width="320" height="191" data-original-width="512" data-original-height="306" /></a></div>
<p>In Sweden this is not the case. Most people go to the doctor when they are sick
and not to prevent you from becoming sick.</p>
<p>When I go to a doctor in Sweden, I would say that they will take the minimum
amount of tests that they think are required. In the US, it felt like the
opposite was true. A lot of tests seemed to be taken that were not strictly
necessary, but they could be motivated and they will profit the hospital and
thus were taken. </p>
<br style="clear:both;">
<p>
This may be good for you as a patient, but it is not good
for society as a whole since it is wasteful and will increase the overall cost.</p>
<h1>Schools and Kids</h1>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhfuuUh28qZ_LMq64JZ6x0jVcGKWQZM7KDdrJ7xamrVKdAlaP2vrCpsNnkl-Z5Xfbcz0hih3h2B9hBYefcljnAYKcmzJY38yBNhJ3mrKZJdcPWfqCYz3izcAniX5RHyrXDTCckd/s1600/henry-m-gunn-high-school.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhfuuUh28qZ_LMq64JZ6x0jVcGKWQZM7KDdrJ7xamrVKdAlaP2vrCpsNnkl-Z5Xfbcz0hih3h2B9hBYefcljnAYKcmzJY38yBNhJ3mrKZJdcPWfqCYz3izcAniX5RHyrXDTCckd/s320/henry-m-gunn-high-school.png" width="320" height="240" data-original-width="538" data-original-height="404" /></a></div>
<p>The schools that my kids attended were awesome. They are ranked as some of the
best public schools in the country and when I compare it to the schools that I
went to when I was young, both in Sweden and in the U.S., it is fantastic. The
students are motivated and so are the teachers. My son, who didn't do much in
school in Sweden, did his homework without any interference from us. Talk about
positive peer pressure. Even though the schools are technically free, you are
expected to give a voluntary donation of $500 per child and year.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhdtyS47kVtJmNK_KfbiLEh9kBpIcygbS3RDoBd4Roo0IjbKfcNqT2kPsukJc5v50toQ0khvstS3yEVeqC4R7I95eiJj5MAam1Ww85ahLHdd5WT4Oz1B0J_r_jyM_dPXujOrs1V/s1600/rasmus-guld.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhdtyS47kVtJmNK_KfbiLEh9kBpIcygbS3RDoBd4Roo0IjbKfcNqT2kPsukJc5v50toQ0khvstS3yEVeqC4R7I95eiJj5MAam1Ww85ahLHdd5WT4Oz1B0J_r_jyM_dPXujOrs1V/s320/rasmus-guld.png" width="320" height="215" data-original-width="480" data-original-height="323" /></a></div>
<p>Sport as part of the school activities is free(ish), but if your kids want
sports outside of school, they have to join a club. The clubs are great, but
expensive. My daughter's diving practice has an Olympic coach from Ukraine,
they practice 6 days a week and it costs us $550 a month and nothing except
training is included! For comparison, her Swedish club, costs about 600 SEK
($60) a year and travel to all competitions is included.</p>
<br style="clear:both;">
<h1>Government</h1>
<p>If you ever been in contact with a U.S. Government you know why so many
Americans are republicans. Any contact with DMV or IRS makes you want to kill
yourself. The only reason there aren't any DMV shootings in the United States
is, because they have a metal detector when you enter their facilities. People
probably go and shoot up the nearest school after their latest Governmental visit.</p>
<p>As if visiting a Governmental office in person is not bad enough, their
telephone service is beyond bad. Imagine a telephone service where you are
supposed to talk to the service instead of pressing buttons. The service is
configured in deaf grand-mother mode and again you are looking for your gun!</p>
<p>Watching the following DMV scene from Zootopia will give you a rough idea, but
reality is worse.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/drhsKTAkRZc" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<h1>Weather</h1>
<p>The weather in the Bay area is typically sunny from the beginning of April to
the end of October. This is quite a treat for a Swede! It's nice to be able to
plan a hike or a BBQ for next week without having to worry about rain!</p>
<p>But, the weather varies a lot from place to place. The temperature in the south
bay can be a lot warmer than the weather on the coast or in San Francisco. I
once drove from 90 degrees (30C) in Saratoga to 60 degrees (15C) in San Francisco.
It taught me to always carry an extra jacket in my car.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQjThsQJKhz4ZI1aKPsiizAhr5ZmAXk-F8rysMxj06rQDFUNlfucLp99fsK0uxPKDFFWbgNSHkio8mMpPrl_6Sn_UETL2ixSWhOGYYwCggEBql8HuM-lolvUiJbz98yICVC0H5/s1600/IMG_20190623_134809.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQjThsQJKhz4ZI1aKPsiizAhr5ZmAXk-F8rysMxj06rQDFUNlfucLp99fsK0uxPKDFFWbgNSHkio8mMpPrl_6Sn_UETL2ixSWhOGYYwCggEBql8HuM-lolvUiJbz98yICVC0H5/s320/IMG_20190623_134809.jpg" width="320" height="240" data-original-width="806" data-original-height="605" /></a></div>
I'll end this with a sunny day at the Sonoma Raceway, you can see the beginnings of the Sonoma and Napa valleys in the background.
Anders Janmyrhttp://www.blogger.com/profile/16081537018884349622noreply@blogger.com0tag:blogger.com,1999:blog-34049130.post-68564376184979425552018-10-22T01:51:00.004+02:002023-04-08T11:06:05.232+02:00Understanding Istio Ingress Gateway in Kubernetes
<p>Traditionally, Kubernetes has used an <code>Ingress</code> controller to handle the
traffic that enters the cluster from the outside. When using Istio, this is no
longer the case. Istio has replaced the familiar <code>Ingress</code> resource with new
<code>Gateway</code> and <code>VirtualServices</code> resources. They work in tandem to route
the traffic into the mesh. Inside the mesh there is no need for <code>Gateway</code>s
since the services can access each other by a cluster local service name.</p>
<p>So how does it work? How does a request reach the application it wants? It is
more complicated than one would think. Here is a drawing and a quick overview.</p>
<ol>
<li>A client makes a request on a specific port.</li>
<li>The <code>Load Balancer</code> listens on this port and forwards the request to one of
the workers in the cluster (on the same or a new port).</li>
<li>Inside the cluster the request is routed to the <code>Istio IngressGateway Service</code>
which is listening on the port the load balancer forwards to.</li>
<li>The <code>Service</code> forwards the request (on the same or a new port) to an <code>Istio
IngressGateway Pod</code> (managed by a Deployment).</li>
<li>The <code>IngressGateway Pod</code> is configured by a <code>Gateway</code> (!) and a <code>VirtualService</code>.</li>
<li>The <code>Gateway</code> configures the ports, protocol, and certificates.</li>
<li>The <code>VirtualService</code> configures routing information to find the correct
<code>Service</code></li>
<li>The <code>Istio IngressGateway Pod</code> routes the request to the <code>application Service</code>.</li>
<li>And finally, the <code>application Service</code> routes the request to an <code>application
Pod</code> (managed by a deployment).</li>
</ol>
<p>
<div class="separator" style="clear: both; text-align: center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigUBWUIRWuF0RVmkKRUqjONp4_JqPYYJd8552c39dXyXF8YCwZkME5CnxFzaAkoZB91GxFVHp7eai52mGII0wOswmmTTxeVmDfSSiGmMhbFK7ewpzmHcHm0BF6pZSzXj5ny4jd/s1600/istio-networking.png" data-original-width="692" data-original-height="489" alt="Routing a Request through Istio Gateway to an Application"/></div/p>
<h2>The Load Balancer</h2>
<p>The load balancer can be configured manually or automatically through the
service <code>type: LoadBalancer</code>. In this case, since not all clouds support
automatic configuration, I'm assuming that the load balancer is configured
manually to forward traffic to a port that the <code>IngressGateway Service</code> is
listening on. Manual load balancers don't communicate with the cluster to find
out where the backing pods are running, and we must expose the Service with
<code>type: NodePort</code> and they are only available on high ports, 30000-32767.
Our LB is listening on the following ports.</p>
<ul>
<li>HTTP - Port 80, forwards traffic to port 30080.</li>
<li>HTTPS - Port 443, forwards traffic to port 30443.</li>
<li>MySQL - Port 3306, forwards traffic to port 30306.</li>
</ul>
<p>Make sure your load balancer configuration forwards to all your worker nodes.
This will ensure that the traffic gets forwarded even if some nodes are down.</p>
<h2>The IngressGateway Service</h2>
<p>The <code>IngressGateway Service</code> must listen to all the above ports to be able to
forward the traffic to the <code>IngressGateway pods</code>. We use the routing to bring
the port numbers back to their default numbers.</p>
<p>Please note that a Kubernetes Service is not a "real" service, but, since we
are using <code>type: NodePort</code>, the request will be handled by the <code>kube-proxy</code>
provided by Kubernetes and forwarded to a node with a running pod. Once on the
node, an IP-tables configuration will forward the request to the appropriate
pod.</p>
<!-- Generator: GNU source-highlight 3.1.8
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt><i><font color="#9A1900"># From the istio-ingressgateway service</font></i>
ports<font color="#990000">:</font>
<font color="#990000">-</font> name<font color="#990000">:</font> http2
nodePort<font color="#990000">:</font> <font color="#993399">30000</font>
port<font color="#990000">:</font> <font color="#993399">80</font>
protocol<font color="#990000">:</font> TCP
<font color="#990000">-</font> name<font color="#990000">:</font> https
nodePort<font color="#990000">:</font> <font color="#993399">30443</font>
port<font color="#990000">:</font> <font color="#993399">443</font>
protocol<font color="#990000">:</font> TCP
<font color="#990000">-</font> name<font color="#990000">:</font> mysql
nodePort<font color="#990000">:</font> <font color="#993399">30306</font>
port<font color="#990000">:</font> <font color="#993399">3306</font>
protocol<font color="#990000">:</font> TCP
</tt></pre>
<p>If you inspect the service, you will see that it defines more ports than I have
describe above. These ports are used for internal Istio communication.</p>
<h2>The IngressGateway Deployment</h2>
<p>Now we have reached the most interesting part in this flow, the
<code>IngressGateway</code>. This is a fancy wrapper around <a href="https://www.envoyproxy.io/">the Envoy
proxy</a> and it is configured in the same way as the
sidecars used inside the service mesh (it is actually the same container). When
we create or change a <code>Gateway</code> or <code>VirtualService</code>, the changes are detected
by the <code>Istio Pilot</code> controller which converts this information to an Envoy
configuration and sends it to the relevant proxies, including the Envoy inside
the <code>IngressGateway</code>.</p>
<p>Don't confuse the <code>IngressGateway</code> with the <code>Gateway</code> resource. The <code>Gateway</code>
resource is used to configure the <code>IngressGateway</code></p>
<p>Since container ports don't have to be declared in Kubernetes pods or
deployments, we don't have to declare the ports in the <code>IngressGateway
Deployment</code>. But, if you look inside the deployment you can see that there are
a number of ports declared anyway (unnecessarily).</p>
<p>What we do have to care about in the <code>IngressGateway Deployment</code> is SSL
certificates. To be able to access the certificates inside the <code>Gateway</code>
resources, make sure that you have <a href="https://istio.io/docs/tasks/traffic-management/secure-ingress/">mounted the certificates
properly</a>.</p>
<!-- Generator: GNU source-highlight 3.1.8
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt><i><font color="#9A1900"># Example certificate volume mounts</font></i>
volumeMounts<font color="#990000">:</font>
<font color="#990000">-</font> mountPath<font color="#990000">:</font> <font color="#990000">/</font>etc<font color="#990000">/</font>istio<font color="#990000">/</font>ingressgateway<font color="#990000">-</font>certs
name<font color="#990000">:</font> ingressgateway<font color="#990000">-</font>certs
readOnly<font color="#990000">:</font> true
<font color="#990000">-</font> mountPath<font color="#990000">:</font> <font color="#990000">/</font>etc<font color="#990000">/</font>istio<font color="#990000">/</font>ingressgateway<font color="#990000">-</font>ca<font color="#990000">-</font>certs
name<font color="#990000">:</font> ingressgateway<font color="#990000">-</font>ca<font color="#990000">-</font>certs
readOnly<font color="#990000">:</font> true
<i><font color="#9A1900"># Example certificate volumes</font></i>
volumes<font color="#990000">:</font>
<font color="#990000">-</font> name<font color="#990000">:</font> ingressgateway<font color="#990000">-</font>certs
secret<font color="#990000">:</font>
defaultMode<font color="#990000">:</font> <font color="#993399">420</font>
optional<font color="#990000">:</font> true
secretName<font color="#990000">:</font> istio<font color="#990000">-</font>ingressgateway<font color="#990000">-</font>certs
<font color="#990000">-</font> name<font color="#990000">:</font> ingressgateway<font color="#990000">-</font>ca<font color="#990000">-</font>certs
secret<font color="#990000">:</font>
defaultMode<font color="#990000">:</font> <font color="#993399">420</font>
optional<font color="#990000">:</font> true
secretName<font color="#990000">:</font> istio<font color="#990000">-</font>ingressgateway<font color="#990000">-</font>ca<font color="#990000">-</font>certs
</tt></pre>
<h2>The Gateway</h2>
<p>The <code>Gateway</code> resources are used to configure the ports for Envoy. Since we have exposed three ports with the service, we need these ports to be handled by Envoy. We can do this by declaring one or more <code>Gateway</code>s. In my example, I'm going to use a single <code>Gateway</code>, but it may be split into two or three.</p>
<!-- Generator: GNU source-highlight 3.1.8
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>apiVersion<font color="#990000">:</font> networking<font color="#990000">.</font>istio<font color="#990000">.</font>io<font color="#990000">/</font>v1alpha3
kind<font color="#990000">:</font> Gateway
metadata<font color="#990000">:</font>
name<font color="#990000">:</font> default<font color="#990000">-</font>gateway
namespace<font color="#990000">:</font> istio<font color="#990000">-</font>system
spec<font color="#990000">:</font>
selector<font color="#990000">:</font>
istio<font color="#990000">:</font> ingressgateway
servers<font color="#990000">:</font>
<font color="#990000">-</font> hosts<font color="#990000">:</font>
<font color="#990000">-</font> <font color="#FF0000">'*'</font>
port<font color="#990000">:</font>
name<font color="#990000">:</font> http
number<font color="#990000">:</font> <font color="#993399">80</font>
protocol<font color="#990000">:</font> HTTP
<font color="#990000">-</font> hosts<font color="#990000">:</font>
<font color="#990000">-</font> <font color="#FF0000">'*'</font>
port<font color="#990000">:</font>
name<font color="#990000">:</font> https
number<font color="#990000">:</font> <font color="#993399">443</font>
protocol<font color="#990000">:</font> HTTPS
tls<font color="#990000">:</font>
mode<font color="#990000">:</font> SIMPLE
privateKey<font color="#990000">:</font> <font color="#990000">/</font>etc<font color="#990000">/</font>istio<font color="#990000">/</font>ingressgateway<font color="#990000">-</font>certs<font color="#990000">/</font>tls<font color="#990000">.</font>key
serverCertificate<font color="#990000">:</font> <font color="#990000">/</font>etc<font color="#990000">/</font>istio<font color="#990000">/</font>ingressgateway<font color="#990000">-</font>certs<font color="#990000">/</font>tls<font color="#990000">.</font>crt
<font color="#990000">-</font> hosts<font color="#990000">:</font> <i><font color="#9A1900"># For TCP routing this fields seems to be ignored, but it is matched</font></i>
<font color="#990000">-</font> <font color="#FF0000">'*'</font> <i><font color="#9A1900"># with the VirtualService, I use * since it will match anything.</font></i>
port<font color="#990000">:</font>
name<font color="#990000">:</font> mysql
number<font color="#990000">:</font> <font color="#993399">3306</font>
protocol<font color="#990000">:</font> TCP
</tt></pre>
<p>Valid ports are, <code>HTTP|HTTPS|GRPC|HTTP2|MONGO|TCP|TLS</code>. More info about
Gateways can be found in the <a href="https://istio.io/docs/reference/config/istio.networking.v1alpha3/#Gateway">Istio Gateway docs</a></p>
<h2>The VirtualService</h2>
<p>Our final interesting resource is the <code>VirtualService</code>, it works in concert
with the Gateway to configure Envoy. If you only add a <code>Gateway</code> nothing will
show up in the Envoy configuration, and the same is true if you only add
a <code>VirtualService</code>.</p>
<p><code>VirtualService</code>s are really powerful and they enable the <a href="https://istio.io/docs/examples/intelligent-routing/">intelligent
routing</a> that is one of the very reasons we want to use Istio in the first place. However, I'm not going into it in this article since it is about the basic networking and not the fancy stuff.</p>
<p>Here's a basic configuration for an HTTP(s) service.</p>
<!-- Generator: GNU source-highlight 3.1.8
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>apiVersion<font color="#990000">:</font> networking<font color="#990000">.</font>istio<font color="#990000">.</font>io<font color="#990000">/</font>v1alpha3
kind<font color="#990000">:</font> VirtualService
metadata<font color="#990000">:</font>
name<font color="#990000">:</font> counter
spec<font color="#990000">:</font>
gateways<font color="#990000">:</font>
<font color="#990000">-</font> default<font color="#990000">-</font>gateway<font color="#990000">.</font>istio<font color="#990000">-</font>system<font color="#990000">.</font>svc<font color="#990000">.</font>cluster<font color="#990000">.</font>local
hosts<font color="#990000">:</font>
<font color="#990000">-</font> counter<font color="#990000">.</font>lab<font color="#990000">.</font>example<font color="#990000">.</font>com
http<font color="#990000">:</font>
<font color="#990000">-</font> match<font color="#990000">:</font>
<font color="#990000">-</font> uri<font color="#990000">:</font>
prefix<font color="#990000">:</font> <font color="#990000">/</font>
route<font color="#990000">:</font>
<font color="#990000">-</font> destination<font color="#990000">:</font>
host<font color="#990000">:</font> counter
port<font color="#990000">:</font>
number<font color="#990000">:</font> <font color="#993399">80</font>
</tt></pre>
<p>Now, when we have added both a <code>Gateway</code> and a <code>VirtualService</code>, the routes
have been created in the Envoy configuration. To see this, you can <code>kubectl
port-forward istio-ingressgateway-xxxx-yyyy 15000</code> and check out the
configuration by browsing to
<a href="http://localhost:15000/config_dump">http://localhost:15000/config_dump</a>.</p>
<p>Note that the gateway specified as well as the host must match the information in the <code>Gateway</code>. If it doesn't the entry will not show up in the configuration.</p>
<!-- Generator: GNU source-highlight 3.1.8
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt><i><font color="#9A1900">// Example of http route in Envoy config</font></i>
<font color="#FF0000">{</font>
name<font color="#990000">:</font> <font color="#FF0000">"counter:80"</font><font color="#990000">,</font>
domains<font color="#990000">:</font> <font color="#990000">[</font>
<font color="#FF0000">"counter.lab.example.com"</font>
<font color="#990000">],</font>
routes<font color="#990000">:</font> <font color="#990000">[</font>
<font color="#FF0000">{</font>
match<font color="#990000">:</font> <font color="#FF0000">{</font>
prefix<font color="#990000">:</font> <font color="#FF0000">"/"</font>
<font color="#FF0000">}</font><font color="#990000">,</font>
route<font color="#990000">:</font> <font color="#FF0000">{</font>
cluster<font color="#990000">:</font> <font color="#FF0000">"outbound|80||counter.default.svc.cluster.local"</font><font color="#990000">,</font>
timeout<font color="#990000">:</font> <font color="#FF0000">"0s"</font><font color="#990000">,</font>
max_grpc_timeout<font color="#990000">:</font> <font color="#FF0000">"0s"</font>
<font color="#FF0000">}</font><font color="#990000">,</font>
<font color="#990000">...</font>
</tt></pre>
<p>Here's a basic configuration for a TCP service.</p>
<!-- Generator: GNU source-highlight 3.1.8
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>apiVersion<font color="#990000">:</font> networking<font color="#990000">.</font>istio<font color="#990000">.</font>io<font color="#990000">/</font>v1alpha3
kind<font color="#990000">:</font> VirtualService
metadata<font color="#990000">:</font>
name<font color="#990000">:</font> mysql
spec<font color="#990000">:</font>
gateways<font color="#990000">:</font>
<font color="#990000">-</font> default<font color="#990000">-</font>gateway<font color="#990000">.</font>istio<font color="#990000">-</font>system<font color="#990000">.</font>svc<font color="#990000">.</font>cluster<font color="#990000">.</font>local
hosts<font color="#990000">:</font> <i><font color="#9A1900"># The host fields seems to only be used to match the Gateway.</font></i>
<font color="#990000">-</font> <font color="#FF0000">'*'</font> <i><font color="#9A1900"># I'm using '*', the listener created is listing on 0.0.0.0</font></i>
tcp<font color="#990000">:</font>
<font color="#990000">-</font> match<font color="#990000">:</font>
<font color="#990000">-</font> port<font color="#990000">:</font> <font color="#993399">3306</font>
route<font color="#990000">:</font>
<font color="#990000">-</font> destination<font color="#990000">:</font>
host<font color="#990000">:</font> mysql<font color="#990000">.</font>default<font color="#990000">.</font>svc<font color="#990000">.</font>cluster<font color="#990000">.</font>local
port<font color="#990000">:</font>
number<font color="#990000">:</font> <font color="#993399">3306</font>
</tt></pre>
<p>This will result in a completely different configuration in the Envoy config.</p>
<!-- Generator: GNU source-highlight 3.1.8
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>listener<font color="#990000">:</font> <font color="#FF0000">{</font>
name<font color="#990000">:</font> <font color="#FF0000">"0.0.0.0_3306"</font><font color="#990000">,</font>
address<font color="#990000">:</font> <font color="#FF0000">{</font>
socket_address<font color="#990000">:</font> <font color="#FF0000">{</font>
address<font color="#990000">:</font> <font color="#FF0000">"0.0.0.0"</font><font color="#990000">,</font>
port_value<font color="#990000">:</font> <font color="#993399">3306</font>
<font color="#FF0000">}</font>
<font color="#FF0000">}</font><font color="#990000">,</font>
</tt></pre>
<h2>Application Service and Deployment</h2>
<p>Our request have now reached the application service and deployment. These are
just normal Kubernetes resources and I will assume that if you have read this
far, you already know all about it. :)</p>
<h2>Debugging</h2>
<p>Debugging networking issues can be difficult at times, so here are some aliases
that I find useful.</p>
<p>Debugging networking issues can be difficult at times, so here are some aliases
that I find useful.</p>
<!-- Generator: GNU source-highlight 3.1.8
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt><i><font color="#9A1900"># Port forward to the first istio-ingressgateway pod</font></i>
<b><font color="#0000FF">alias</font></b> <font color="#009900">igpf</font><font color="#990000">=</font><font color="#FF0000">'kubectl -n istio-system port-forward $(kubectl -n istio-system</font>
<font color="#FF0000">get pods -listio=ingressgateway -o=jsonpath="{.items[0].metadata.name}") 15000'</font>
<i><font color="#9A1900"># Get the http routes from the port-forwarded ingressgateway pod (requires jq)</font></i>
<b><font color="#0000FF">alias</font></b> <font color="#009900">iroutes</font><font color="#990000">=</font><font color="#FF0000">'curl --silent http://localhost:15000/config_dump |</font>
<font color="#FF0000">jq '</font>\'<font color="#FF0000">'.configs.routes.dynamic_route_configs[].route_config.virtual_hosts[]|</font>
<font color="#FF0000">{name: .name, domains: .domains, route: .routes[].match.prefix}'</font>\'<font color="#FF0000">''</font>
<i><font color="#9A1900"># Get the logs of the first istio-ingressgateway pod</font></i>
<i><font color="#9A1900"># Shows what happens with incoming requests and possible errors</font></i>
<b><font color="#0000FF">alias</font></b> <font color="#009900">igl</font><font color="#990000">=</font><font color="#FF0000">'kubectl -n istio-system logs $(kubectl -n istio-system get pods</font>
<font color="#FF0000">-listio=ingressgateway -o=jsonpath="{.items[0].metadata.name}") --tail=300'</font>
<i><font color="#9A1900"># Get the logs of the first istio-pilot pod</font></i>
<i><font color="#9A1900"># Shows issues with configurations or connecting to the Envoy proxies</font></i>
<b><font color="#0000FF">alias</font></b> <font color="#009900">ipl</font><font color="#990000">=</font><font color="#FF0000">'kubectl -n istio-system logs $(kubectl -n istio-system get pods</font>
<font color="#FF0000">-listio=pilot -o=jsonpath="{.items[0].metadata.name}") discovery --tail=300'</font>
</tt></pre>
<p>When you have started the port-forwarding to the <code>istio-ingressgateway</code>, with
<code>igpf</code>, here are some more things you can do.</p>
<ul>
<li>To see the Envoy listeners, browse to <a href="http://localhost:15000/listeners">http://localhost:15000/listeners</a>.</li>
<li>To turn on more verbose logging, browse to <a href="http://localhost:15000/logging">http://localhost:15000/logging</a>.</li>
<li>More information can be found at the root, <a href="http://localhost:15000/">http://localhost:15000/</a>.</li>
</ul>
<h2>Conclusion</h2>
<p>Networking with Kubernetes and Istio is far from trivial, hopefully this
article has shed some light on how it works. Here are some key takeaways.</p>
<h3>To Add a New Port to the IngressGateway</h3>
<ul>
<li>Add the port to an existing <code>Gateway</code> or configure a new.</li>
<li>If it's a TCP service also add the port to the <code>VirtualService</code>, not needed
for HTTP since it matches on layer 7 (domain name, etc.).</li>
<li>Add the port to the <code>ingressgateway service</code>. If you are using service
<code>type: LoadBalancer</code>, you are done.</li>
<li>Otherwise, open the port in the load balancer and forward traffic to all
worker nodes.</li>
</ul>
<h3>To Add Certificates to an SSL Service</h3>
<ul>
<li>Add the TLS secrets to the cluster.</li>
<li>Mount the secret volumes in the <code>ingressgateway</code>.</li>
<li>Configure the <code>Gateway</code> to use the newly created secrets.</li>
</ul>
Anders Janmyrhttp://www.blogger.com/profile/16081537018884349622noreply@blogger.com0tag:blogger.com,1999:blog-34049130.post-67729225974600389252017-03-12T22:29:00.001+01:002017-03-12T22:30:12.733+01:00A Short Introduction to Makefiles<p>Makefiles are really good at one thing, managing dependencies between files. In
other words, <code>make</code> makes sure all files that depend on another file are
updated when that file changes.</p>
<p>We tell <code>make</code> how to do this by declaring rules. A typical rule looks like
this:</p>
<h2>A Simple Makefile</h2>
<blockquote><pre><tt><i><font color="#9A1900"># Makefile</font></i>
<i><font color="#9A1900"># Create bundle.js by concatenating jquery.js lib.js and main.js</font></i>
<font color="#990000">bundle.js:</font> jquery.js lib.js main.js
cat <font color="#009900">$^</font> <font color="#990000">></font> <font color="#009900">$@</font>
</tt></pre></blockquote>
<p>There are three parts to this rule:</p>
<ul>
<li>The target, <code>bundle.js</code>, before the colon (:).</li>
<li>The prerequisites (what the target depends on), <code>jquery.js lib.js main.js</code>,
after the colon (:).</li>
<li>The command, <em>cat $^ > $@</em>, on the next line <strong>after a leading tab, (\t)</strong>.</li>
</ul>
<p>There are two "automatic" variables in this command.</p>
<ul>
<li>$@ - filename representing the target, in this case <code>bundle.js</code>.</li>
<li>$^ - filenames representing the list of the prerequisites (with duplicates
removed).</li>
</ul>
<p>"Automatic" means that the variables are automatically populated with relevant
filenames. This will <code>make</code> more sense when we get into patterns later.</p>
<p>Here are some more variables that are useful.</p>
<ul>
<li>$< - filename representing the first prerequisite.</li>
<li>$? - filenames representing the list of the prerequisites that are newer
than the target.</li>
<li><code>$*</code> - filename representing the stem of the target, in the above case <code>bundle</code>.</li>
</ul>
<p>The Make Manual contains <a href="https://www.gnu.org/software/make/manual/make.html#Automatic-Variables">the full list of automatic
variables</a></p>
<h2>Execution</h2>
<p>Running <code>make</code> with the above Makefile results in the following execution.</p>
<blockquote><pre><tt>$ make
cat jquery<font color="#990000">.</font>js lib<font color="#990000">.</font>js main<font color="#990000">.</font>js <font color="#990000">></font> bundle<font color="#990000">.</font>js
$ make
make<font color="#990000">:</font> <font color="#FF0000">'bundle.js'</font> is up to date<font color="#990000">.</font>
</tt></pre></blockquote>
<p><code>make</code> runs the first target it finds in the file if none is given on the
command line. In this case it is the only target.</p>
<p>The second run didn't do anything since <code>bundle.js</code> is up to date. To be <em>up to
date</em> means that the <code>last-modified</code> time of <code>bundle.js</code> is newer than any of
its prerequisites' <code>last-modified</code> times. Simple but powerful! When we create
new targets all we have to worry about is making sure that our targets know
what files it depends on, and what files they depend on, and so on.</p>
<p>It is possible to enter many targets on the left of the colon (:). <code>make</code> will
treat them as separate rules and the automatic variable will make sure that the
correct files are built.</p>
<p>But, since <code>make</code> treats the rules as separate rules, it will only build the
first of them, the default target.</p>
<blockquote><pre><tt><i><font color="#9A1900"># Makefile</font></i>
<font color="#990000">bundle.js bundle2.js:</font> jquery.js lib.js main.js
cat <font color="#009900">$^</font> <font color="#990000">></font> <font color="#009900">$@</font>
</tt></pre></blockquote>
<blockquote><pre><tt>$ make
make<font color="#990000">:</font> <font color="#FF0000">'bundle.js'</font> is up to date<font color="#990000">.</font>
</tt></pre></blockquote>
<p>If we want to build <code>bundle2.js</code>, we can do it by explicitly telling <code>make</code> to do
it by giving the target as command line parameter.</p>
<blockquote><pre><tt>$ make bundle2<font color="#990000">.</font>js
cat jquery<font color="#990000">.</font>js lib<font color="#990000">.</font>js main<font color="#990000">.</font>js <font color="#990000">></font> bundle2<font color="#990000">.</font>js
</tt></pre></blockquote>
<p>To get <code>make</code> to build both targets at once, we need to add a new, <code>.PHONY:</code>,
target.</p>
<blockquote><pre><tt><i><font color="#9A1900"># Makefile</font></i>
<b><font color="#000080">.PHONY:</font></b> bundles
<font color="#990000">bundles:</font> bundle.js bundle2.js
<font color="#990000">bundle.js bundle2.js:</font> jquery.js lib.js main.js
cat <font color="#009900">$^</font> <font color="#990000">></font> <font color="#009900">$@</font>
</tt></pre></blockquote>
<p>Running <code>make</code> now results in (after removing bundle*)</p>
<blockquote><pre><tt>$ make
cat jquery<font color="#990000">.</font>js lib<font color="#990000">.</font>js main<font color="#990000">.</font>js <font color="#990000">></font> bundle<font color="#990000">.</font>js
cat jquery<font color="#990000">.</font>js lib<font color="#990000">.</font>js main<font color="#990000">.</font>js <font color="#990000">></font> bundle2<font color="#990000">.</font>js
</tt></pre></blockquote>
<p>A <code>.PHONY:</code> target is a target without a corresponding file for <code>make</code> to check
last modified time on. This means the target will always be run, forcing <code>make</code>
to check if all the target's prerequisites needs to be built. The <code>.PHONY</code>
label is not strictly necessary. If it is left out, <code>make</code> will check to see if
there is a file called <code>bundles</code> and since there isn't one it will build it
anyway.</p>
<p>Here's an illustration:</p>
<blockquote><pre><tt><i><font color="#9A1900"># Makefile</font></i>
<font color="#990000">build:</font>
echo <font color="#FF0000">'Running build'</font>
</tt></pre></blockquote>
<blockquote><pre><tt><i><font color="#9A1900"># Makefile2</font></i>
.PHONY
<font color="#990000">build:</font>
echo <font color="#FF0000">'Running build'</font>
</tt></pre></blockquote>
<blockquote><pre><tt>$ touch build
$ make
make<font color="#990000">:</font> <font color="#FF0000">'build'</font> is up to date<font color="#990000">.</font>
$ make -f Makefile2
echo <font color="#FF0000">'Running build'</font>
</tt></pre></blockquote>
<p>Marking a target that doesn't represent a file as <code>.PHONY:</code> is easy to do and
avoids annoying problems once your <code>Makefile</code> grows.</p>
<h3>.PHONY: clean</h3>
<p>Conventionally every <code>Makefile</code> contains a <code>clean</code> target to remove all the
artifacts that are built. In the above case it would contain something like:</p>
<blockquote><pre><tt><i><font color="#9A1900"># Makefile</font></i>
<b><font color="#000080">.PHONY:</font></b> clean
<font color="#990000">clean:</font>
rm -f bundle<font color="#990000">*</font>.js
</tt></pre></blockquote>
<p><code>make clean</code> will now clean out all files created by the Makefile.</p>
<h3>Directories</h3>
<p>Directories in <code>make</code> usually needs a bit of special treatment. Let's say we want
the bundles above to end up in a <code>build</code> directory. The following <code>Makefile</code>
illustrates a problem.</p>
<blockquote><pre><tt><i><font color="#9A1900"># Makefile</font></i>
<font color="#990000">bundles:</font> build/bundle.js build/bundle2.js
build/bundle.js build/bundle2.js<font color="#990000">:</font> jquery.js lib.js main.js
cat <font color="#009900">$^</font> <font color="#990000">></font> <font color="#009900">$@</font>
</tt></pre></blockquote>
<p>Running <code>make</code> illustrates the problem:</p>
<blockquote><pre><tt>cat jquery<font color="#990000">.</font>js lib<font color="#990000">.</font>js main<font color="#990000">.</font>js <font color="#990000">></font> build/bundle<font color="#990000">.</font>js
/bin/sh<font color="#990000">:</font> build/bundle<font color="#990000">.</font>js<font color="#990000">:</font> No such file or directory
make<font color="#990000">:</font> <font color="#990000">***</font> <font color="#990000">[</font>build/bundle<font color="#990000">.</font>js<font color="#990000">]</font> Error <font color="#993399">1</font>
</tt></pre></blockquote>
<p>The directory is not automatically created by <code>cat</code>. There are three ways to
solve this and one is better than the others.</p>
<ol>
<li>Add <code>mkdir -p</code> to all rules creating files in the directory.</li>
<li>Add a prerequisite to create the directory on the <code>bundles</code> target.</li>
<li>Add an ordering prerequisite (|) to the rules creating the files in the
directory.</li>
</ol>
<p><code>1.</code> is not good because the directory will be created more than once, one for
each bundle (this is why the <code>-p</code> is needed).
<code>2.</code> is not good because the <code>build</code> directory is not a prerequisite target
for <code>bundles</code>.
<code>3.</code> is good because the <code>build</code> directory is clearly a prerequisite of the rule
that creates the bundles in this directory.</p>
<p>The reason we have to use an ordering prerequisite instead of a normal
prerequisite is that <code>cat</code> would fail otherwise. Here's the resulting good
<code>Makefile</code>.</p>
<blockquote><pre><tt><i><font color="#9A1900"># Makefile</font></i>
<font color="#990000">bundles:</font> build build/bundle.js build/bundle2.js
<font color="#990000">build:</font>
mkdir build
build/bundle.js build/bundle2.js<font color="#990000">:</font> jquery.js lib.js main.js <font color="#990000">|</font> build
cat <font color="#009900">$^</font> <font color="#990000">></font> <font color="#009900">$@</font>
<font color="#990000">clean:</font>
rm -rf build
</tt></pre></blockquote>
<h2>Patterns</h2>
<p>Now, we know the basics of Makefiles. We can create rules with targets,
prerequisites and commands that are run when needed. But, we have been working
with named files all this time. This works fine for small examples like above,
but when we have hundreds of files this quickly gets out of hand. Patterns to
the rescue.</p>
<p>Let's say that we have a bunch of images that we would like to optimize by
running them through an optimizer. The images are in the <code>images/</code> directory
and the optimized images are built into <code>build/images</code>. The naive (and not
working) way to do this is shown below. (I'm faking optimize with a simple
copy, <code>cp</code>.) The <code>%</code> sign is glob matched with the part of the filename that is
not literal.</p>
<blockquote><pre><tt><i><font color="#9A1900"># Makefile (NOT WORKING)</font></i>
<font color="#990000">optimize:</font> build/images<font color="#990000">/*</font>
build/images<font color="#990000">/%:</font> images<font color="#990000">/%</font> <font color="#990000">|</font> build/images
cp <font color="#009900">$<</font> <font color="#009900">$@</font>
build/images<font color="#990000">:</font>
mkdir -p <font color="#009900">$@</font>
</tt></pre></blockquote>
<p>To see why this is not a viable Makefile, we try to run it with <code>make</code>.</p>
<blockquote><pre><tt>$ make
mkdir -p build/images
cp images/a<font color="#990000">.</font>png build/images<font color="#990000">/*.</font>
$ tree build
build<font color="#990000">/</font>
└── images
└── <font color="#990000">*</font>
</tt></pre></blockquote>
<p>What is going on here? Why is only one image copied and why is it copied as
name <code>build/images/*</code>? The problem is that the target files don't exist yet and
the <code>*</code> is interpreted literally. If we copy the files into the build directory
and touch the source files, it works the way we want.</p>
<blockquote><pre><tt><i><font color="#9A1900"># Copy the image directory into build</font></i>
$ cp -r images build
<i><font color="#9A1900"># Touch the orignal images</font></i>
$ touch images<font color="#990000">/*</font>
<i><font color="#9A1900"># Build works since build/image/* evaluates to the list of images</font></i>
$ make
cp images/a<font color="#990000">.</font>png build/images/a<font color="#990000">.</font>png
cp images/b<font color="#990000">.</font>png build/images/b<font color="#990000">.</font>png
</tt></pre></blockquote>
<p>Here is the main rule to know about patterns. <strong>The target file list has to
be created from the available source files.</strong> To do this we have help of a
number of functions, including <code>wildcard</code>, <code>shell</code>, etc. <code>shell</code> will allow us
to call anything that we can call from the shell This is very powerful!</p>
<p>How do we solve the above problem? We can do this by getting a list of source
images and transforming this list into a list of target images. This is easily
done.</p>
<blockquote><pre><tt><i><font color="#9A1900"># Makefile</font></i>
<i><font color="#9A1900"># 1. Get the souce list of images</font></i>
<font color="#990000">images :=</font> <font color="#009900">$(</font>wildcard images<font color="#990000">/*</font>.png<font color="#990000">)</font>
<i><font color="#9A1900"># 2. Tranform the source list into the target list</font></i>
<font color="#990000">target_images :=</font> <font color="#009900">$(images:%=build/%)</font>
<i><font color="#9A1900"># 3. Our default target, optimize, depends on all the target_images</font></i>
<font color="#990000">optimize:</font> <font color="#009900">$(target_images)</font>
<i><font color="#9A1900"># 4. Build the targets from the sources, make sure build/images exist</font></i>
<font color="#009900">$(target_images)</font><font color="#990000">:</font> build<font color="#990000">/%</font> <font color="#990000">:</font> <font color="#990000">%</font> <font color="#990000">|</font> build/images
cp <font color="#009900">$<</font> <font color="#009900">$@</font>
build/images<font color="#990000">:</font>
mkdir -p <font color="#009900">$@</font>
</tt></pre></blockquote>
<p>The first line introduces both variables and functions.</p>
<p>Variables can be declared in <a href="https://www.gnu.org/software/make/manual/make.html#Setting">a number of
ways</a>, but the
<code>:=</code>-declaration is the simplest. It evaluates the value on the right and sets
the value on the left to the result, like variables in most programming
languages.</p>
<p>Functions are called with the <code>$()</code> construct, and <code>wildcard</code> is a function
that evaluates a <em>shell</em> filename pattern and returns a list of filenames.</p>
<p>The full line above populates <code>images</code> with the <code>.png</code> files from the <code>images</code>
directory.</p>
<p>The second line converts the source images into the target images. Variables
are evaluated the same way as function calls, with the <code>$()</code> construct. By
adding a colon-equals expression, a <em>variable substitution reference</em>, after the
variable name we can substitute a pattern for another. Example</p>
<blockquote><pre><tt><font color="#990000">files :=</font> <font color="#FF0000">"src/a.java src/b.java"</font>
<i><font color="#9A1900"># Pattern replaces the files into "target/a.class target/b.class"</font></i>
<font color="#990000">class_files :=</font> <font color="#009900">$(files:src/%.java:target/%.class)</font>
</tt></pre></blockquote>
<p>The third line tells <code>make</code> that our <code>optimize</code> target depends on all the
targets existing. This makes sure that all the targets are built.</p>
<p>The fourth line sets up the targets `$(target_images) and its prerequisites
with a <a href="https://www.gnu.org/software/make/manual/make.html#Static-Pattern">static pattern rule</a>. The pattern does the opposite of the variable substitution reference above, it
deconstructs a single target into the source it depends on. The final part of
the of this line is an <em>order prerequisite</em> on the rule to create the
directory.</p>
<h2>A Recipe for Creating Makefiles</h2>
<ul>
<li>Create a list of targets that you want to create from the sources. You have
the full power of <code>bash</code>, <code>python</code>, <code>awk</code>, etc. at your disposal.</li>
<li>Create a static pattern rule to convert a single target into the source it
can be created from.</li>
<li>Add <em>order prerequisites</em> to make sure directories are automatically created.</li>
<li>Add a callable target that depends on all the target files that you want to
create.</li>
</ul>
<h3>Commented Example</h3>
<p>Here's a more exotic example of what you can use <code>make</code> for. We have a directory
of Javascript source files in <code>lib</code>. The corresponding test files are in
<code>test</code>. There may be multiple directories below both <code>lib</code> and <code>test</code>. The
testfiles are named like the source files with an added <code>.spec</code> after the stem
of the filename.</p>
<p>We want to use a makefile to help us run only the tests that are relevant based
on the files that are changed. To keep track of what tests have been run we're
going to use marker files and <code>touch</code> them every time a test is run.</p>
<blockquote><pre><tt><i><font color="#9A1900"># Create the list of test files by using the shell function and find</font></i>
<font color="#990000">test_files :=</font> <font color="#009900">$(</font>shell find <b><font color="#0000FF">test</font></b> -name <font color="#FF0000">'*.spec.js'</font> -print<font color="#990000">)</font>
<i><font color="#9A1900"># Convert the test files into marker files with variable substitution</font></i>
<i><font color="#9A1900"># A marker files looks like this tmp/model/person_test.marker</font></i>
<font color="#990000">marker_files :=</font> <font color="#009900">$(test_files:%.js=tmp/%.marker)</font>
<i><font color="#9A1900"># Do the same thing for the test directories</font></i>
<font color="#990000">test_dirs :=</font> <font color="#009900">$(</font>shell find <b><font color="#0000FF">test</font></b> -type d -print<font color="#990000">)</font>
<i><font color="#9A1900"># The marker directories have their normal names, no special ending</font></i>
<font color="#990000">marker_dirs :=</font> <font color="#009900">$(test_dirs:%=tmp/%)</font>
<i><font color="#9A1900"># test is the default target</font></i>
<b><font color="#000080">.PHONY:</font></b> <b><font color="#0000FF">test</font></b>
<font color="#990000">test:</font> <font color="#009900">$(marker_files)</font>
<i><font color="#9A1900"># The marker files order depend on the marker directories</font></i>
<font color="#009900">$(marker_files)</font><font color="#990000">:</font> <font color="#990000">|</font> <font color="#009900">$(marker_dirs)</font>
<i><font color="#9A1900"># Marker files depend on the source files</font></i>
<i><font color="#9A1900"># Deconstruct a marker file into a source file</font></i>
<font color="#009900">$(marker_files)</font><font color="#990000">:</font> tmp/test<font color="#990000">/%</font>.spec.marker <font color="#990000">:</font> lib<font color="#990000">/%</font>.js
<i><font color="#9A1900"># Marker files depend on the test files</font></i>
<i><font color="#9A1900"># Deconstruct the marker file into a test file</font></i>
<i><font color="#9A1900"># When any prerequisite changes, run the tests and then touch the marker file</font></i>
<font color="#009900">$(marker_files)</font><font color="#990000">:</font> tmp<font color="#990000">/%</font>.marker <font color="#990000">:</font> <font color="#990000">%</font>.js
mocha <font color="#009900">$<</font>
<b><font color="#000080">@touch $@</font></b>
<i><font color="#9A1900"># Create the marker dirs</font></i>
<font color="#009900">$(marker_dirs)</font><font color="#990000">:</font>
<b><font color="#000080">@mkdir -p $@</font></b>
<i><font color="#9A1900"># Clean the project by removing the entire tmp directory</font></i>
<b><font color="#000080">.PHONY:</font></b> clean
<font color="#990000">clean:</font>
rm -rf tmp
</tt></pre></blockquote>
<p>Now whenever you run <code>make</code>, it will run only the relevant tests.</p>
<blockquote><pre><tt><i><font color="#9A1900"># Modify a test file</font></i>
$ touch test/models/passbook<font color="#990000">.</font>spec<font color="#990000">.</font>js
$ make
mocha test/models/passbook<font color="#990000">.</font>spec<font color="#990000">.</font>js
passbook
✓ generate pass strips image names
✓ doesnt crash with no store number
<font color="#993399">2</font> passing <font color="#990000">(</font>21ms<font color="#990000">)</font>
<i><font color="#9A1900"># Modify a source file</font></i>
$ touch lib/models/passbook<font color="#990000">.</font>js
$ make
mocha test/models/passbook<font color="#990000">.</font>spec<font color="#990000">.</font>js
passbook
✓ generates pass strip image names
✓ doesnt crash with no store number
<font color="#993399">2</font> passing <font color="#990000">(</font>22ms<font color="#990000">)</font>
</tt></pre></blockquote>
<p>Makefiles are really good at one thing: <strong>building only stale files</strong>. If that
is our problem, we should give <code>make</code> a try.</p>
Anders Janmyrhttp://www.blogger.com/profile/16081537018884349622noreply@blogger.com0tag:blogger.com,1999:blog-34049130.post-11063688305893028512016-03-09T09:45:00.000+01:002016-03-09T09:45:05.370+01:00Programming an HS-1969
<p>We are all programmers even you who don't consider yourselves programmers.
We are programmers of the hardest computer of all, the Homo Sapiens, ourselves!</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-E42CBGwH2WHt8hitVioTe4QkAPFy3fIAWJdfmh9ODzMkkRukqOt3srqyKxoBChpr3Jk3SuQZks0nPw_pdjzp1uQSR01hk98Z-l5ULY9ZPkxNUHzjVqxvVk8t0Ccvd7Cb0QFc/s1600/programming-an-hs1969-images.002.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-E42CBGwH2WHt8hitVioTe4QkAPFy3fIAWJdfmh9ODzMkkRukqOt3srqyKxoBChpr3Jk3SuQZks0nPw_pdjzp1uQSR01hk98Z-l5ULY9ZPkxNUHzjVqxvVk8t0Ccvd7Cb0QFc/s200/programming-an-hs1969-images.002.png" /></a></div>
<h2>Aha</h2>
<p>14 years ago my son Rasmus was born. It was a difficult time, partly because he
had colic, but mostly because I couldn't understand what this little critter
wanted. But, one day while I was changing his diaper he said something that
sounded like Aha! Aha, I thought out loud and he reacted with a smile and said
Aha, again. This little kid had picked up that every once in a while I actually
understood what he was trying to communicate and when I did I said Aha! He
liked it so much that he learned to say Aha himself almost before he could say
anything else.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgE66V6nhloJmEcLPxM7rTI3-ryGoXG1tWn5GpgUEOzwJ9trLMT4-Gh1TRZuD9-JUn0oY53LNWozQfQS3KvxyefVhByIXIjOIf-2GZ64jZ1Muvuxg9v-C9G-s4nZ7uKL2l3c1Fn/s1600/programming-an-hs1969-images.004.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgE66V6nhloJmEcLPxM7rTI3-ryGoXG1tWn5GpgUEOzwJ9trLMT4-Gh1TRZuD9-JUn0oY53LNWozQfQS3KvxyefVhByIXIjOIf-2GZ64jZ1Muvuxg9v-C9G-s4nZ7uKL2l3c1Fn/s200/programming-an-hs1969-images.004.png" /></a></div>
<h2>Insight</h2>
<p>We all know what an insight is and that it is a great feeling. Having an
Aha-moment feels great! A lot of human progress (all of it?) has its roots in
Aha moments.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjxgToutJhixQIZVBWsE9caKH0d9a3XyP0i8eVD5GvBn-kDsr9dkGCtg9GVNbBOTkz5T5_B74tAjjM3Ty0fk_UZ1mszn28v0Mr6yW66zQ-v7VLdfRZkeVZ5x9tEDIEootDmWGa8/s1600/programming-an-hs1969-images.005.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjxgToutJhixQIZVBWsE9caKH0d9a3XyP0i8eVD5GvBn-kDsr9dkGCtg9GVNbBOTkz5T5_B74tAjjM3Ty0fk_UZ1mszn28v0Mr6yW66zQ-v7VLdfRZkeVZ5x9tEDIEootDmWGa8/s200/programming-an-hs1969-images.005.png" /></a></div>
<h3>Einstein</h3>
<p>
Einstein is known as a very intuitive scientist. He had a lot of insights and
it was not only the special theory of relativity. He proposed the quantum theory
of light and the link between mass and energy and got the nobel prize for the photoelectric
effect.
</p>
<p>
When he was shaving in the
mornings he always shaved very slowly, because he often had Aha-moments while
shaving and was afraid to cut himself with the razor.</p>
<h3>Examples of Insights</h3>
<p>An insight can help with a lot of things:</p>
<ul>
<li>The punchline of a joke</li>
<li>The solution to a crossword puzzle, riddle or rebus.</li>
<li>Understanding why people behave the way they do.</li>
<li>Resolving inconsistencies in our thinking.</li>
<li>Realizing that one problem is similar to another.</li>
</ul>
<h3>Understanding</h3>
<p>More than anything an insight is understanding. It is when the pieces fall
together and we finally get how something or someone works.</p>
<p>A Doh moment is also an Aha-moment, but it is when you are realizing something
trivial that you believe you should have known all along. An example can be a
song you have been singing your whole life and suddenly you realize that you
have misunderstood the song all along. Here is a personal example:</p>
<pre><code>Jimmy Hendrix sings: Excuse me while I kiss the sky,
not: Excuse me while I kiss this guy.
</code></pre>
<p>Even though the latter would have been more progressive :)</p>
<h3>Some Insight Problems</h3>
<p>What historical person does <strong>"Horobod"</strong> symbolize?</p>
<p>A window washer fell from a 40-foot ladder without hurting himself. How is this
possible?</p>
<p>Thiss sentence has thre errors? What are they?</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiIvrRXSF9oolVXzoiR79kiZ90kGsvtW9srnVBK-E27t8Ea_j7cwlNjRl3079FjJIAuNA3IpfdX1GORAx9duaDdb9luRkjukcFL7V-_4SR781iAiJekn5IB2Hmgq4oQUQiZqngf/s1600/programming-an-hs1969-images.012.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiIvrRXSF9oolVXzoiR79kiZ90kGsvtW9srnVBK-E27t8Ea_j7cwlNjRl3079FjJIAuNA3IpfdX1GORAx9duaDdb9luRkjukcFL7V-_4SR781iAiJekn5IB2Hmgq4oQUQiZqngf/s200/programming-an-hs1969-images.012.png" /></a></div>
<h3>Fermat's Conjecture</h3>
<p>Fermat conjectured his theorem in 1637 in the margin of Arithmetica. He wrote,</p>
<pre><code>
It is impossible to separate a cube into two cubes
or a fourth power into two fourth powers, or in general
any power higher than the second into two like powers.
I have discovered a truly marvelous proof of this,
which this margin is too narrow to contain.
</code></pre>
<p>It took 358 years for mankind to come up with this proof and Andrew Wiles spent
7 years of his research time to prove it. Here is how he describes it:</p>
<iframe width="480" height="360" src="https://www.youtube.com/embed/SccDUpIPXM0?rel=0" frameborder="0" allowfullscreen></iframe>
<h3>Two More Insight Problems</h3>
<p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhajurVyT0lmjDe0TU-LlebamLfcfuSELtOEd9zZ6AQHhPoZdEBn4vOKpafIrhfJ3-Dduz6-DA5ivCaXclMX_bRMQPBY7KABUnrjgfWo6cYIIqIyQ8VgzX2h3QWLstXxZZrzraw/s1600/programming-an-hs1969-images.014.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhajurVyT0lmjDe0TU-LlebamLfcfuSELtOEd9zZ6AQHhPoZdEBn4vOKpafIrhfJ3-Dduz6-DA5ivCaXclMX_bRMQPBY7KABUnrjgfWo6cYIIqIyQ8VgzX2h3QWLstXxZZrzraw/s200/programming-an-hs1969-images.014.png" /></a></div>
Turn the pyramid upside down by moving three coins.
</p>
<p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj6EnApxXrSFKubbQmD73Yi-fl1-WvomGJGH1AKwB12EEvYtYD-VUS_ANE9iL5XleQ4UIzAxkvfyJWpcqwBSpRoV9iCduWrEuaPVWrqEKJllaLbx6qmrSKNkQE4qN0sE2Pjy9p2/s1600/programming-an-hs1969-images.015.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj6EnApxXrSFKubbQmD73Yi-fl1-WvomGJGH1AKwB12EEvYtYD-VUS_ANE9iL5XleQ4UIzAxkvfyJWpcqwBSpRoV9iCduWrEuaPVWrqEKJllaLbx6qmrSKNkQE4qN0sE2Pjy9p2/s200/programming-an-hs1969-images.015.png" /></a></div>
Draw four straight lines through all the dots without lifting the pen.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj-d22QNiZw6hXiFdnZptOZee7GwiGFd77Rd086e68G1V2YdOO8qoHo61QQQdIQuoyz8chrATkw2lAbe1L69g1X9qxUzRHHJL-YDVQMjVgLOUDBFbFZz3ZscDMX4r_bAjhQbjFm/s1600/programming-an-hs1969-images.016.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj-d22QNiZw6hXiFdnZptOZee7GwiGFd77Rd086e68G1V2YdOO8qoHo61QQQdIQuoyz8chrATkw2lAbe1L69g1X9qxUzRHHJL-YDVQMjVgLOUDBFbFZz3ZscDMX4r_bAjhQbjFm/s200/programming-an-hs1969-images.016.png" /></a></div>
<h3>When Do We Get Insights? Bath, Bed, and Bus</h3>
<p>Insights can come at any time, but most often when we are not actively focusing
on the problem. Wittgenstein famously said that "the key to creative thinking
is the three B:s, Bath, Bed, and Bus.</p>
<p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhqBt-B7vjVapxsWwcsE_z7DAh4Xv_Lr8_XKX3EHKlsZXIkdHibJM_NJZwkkl05PGo6i32vCaDxYGn-2VHaHJJ9jXjyQDG_JDhTpc5rdtsiqqajwSdbPL5TLFetiIZSATPgMZbv/s1600/programming-an-hs1969-images.017.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhqBt-B7vjVapxsWwcsE_z7DAh4Xv_Lr8_XKX3EHKlsZXIkdHibJM_NJZwkkl05PGo6i32vCaDxYGn-2VHaHJJ9jXjyQDG_JDhTpc5rdtsiqqajwSdbPL5TLFetiIZSATPgMZbv/s200/programming-an-hs1969-images.017.png" /></a></div>
Archimedes got his Eureka-moment when he was lowering himself into a bath and
realized that he could measure the volume of an irregular body by lowering it
into water and measuring the amount of water that flowed out.</p>
<p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAScP60LBnhRm0VmyMDKFLCx5DP7wN048KItkCrlRgolMeZAQI-ZlBXrLeoNRXqZeuQ2Zdm_Xw4wEB5yYyy7iIV8npu7p-dR5YalAL9pHj1LjBnwtxizHAPJvmGMrvUiWQyZKh/s1600/programming-an-hs1969-images.018.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAScP60LBnhRm0VmyMDKFLCx5DP7wN048KItkCrlRgolMeZAQI-ZlBXrLeoNRXqZeuQ2Zdm_Xw4wEB5yYyy7iIV8npu7p-dR5YalAL9pHj1LjBnwtxizHAPJvmGMrvUiWQyZKh/s200/programming-an-hs1969-images.018.png" /></a></div>
Edison used to take working naps. He would sit down on a chair with two metal
balls in his hands and rest. When he fell asleep the balls would fall out of
his hands and onto the steel pan he had placed below himself. He was trying to
trigger insights which he often got when falling asleep.</p>
<p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgZo0RTPbrqxlhpcBFWvlEjyrGA5BNLvtAt4LsoIvSL0QOna1JggGCSHyu8B69xVCVooMTOKvAhAbFb78olgn0OMknxlBBpiN65tSEBaIj19VWTLMMWAlprorLkLQwfiS-9EUT9/s1600/programming-an-hs1969-images.019.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgZo0RTPbrqxlhpcBFWvlEjyrGA5BNLvtAt4LsoIvSL0QOna1JggGCSHyu8B69xVCVooMTOKvAhAbFb78olgn0OMknxlBBpiN65tSEBaIj19VWTLMMWAlprorLkLQwfiS-9EUT9/s200/programming-an-hs1969-images.019.png" /></a></div>
Poincaré got one of his greatest ideas when he was about to get on a bus.</p>
<pre><code>
At the moment when I put my foot on the
step the idea came to me, without
anything in my former thoughts seeming to
have paved the way for it, that the
transformation that I had used to define
the Fuchsian functions were identical
with those of non-euclidean geometry.
</code></pre>
<h3>How do we get insights?</h3>
<p>In order to get insight about something you have to prepare for it. It is
impossible to get insights into something you know nothing about.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiXiTqsSexOddq3PtmH1DMrMCnlv2nVrPQST5pHXUmdVAzYSVOvZeWPPg4CaV3WgEm7VOJ04qy0QKggK7eWSKpnCvDKlxkK9U2mB9Nmd4qipHuDu25t27CE9dAHX1gI-ebtniiz/s1600/programming-an-hs1969-images.021.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiXiTqsSexOddq3PtmH1DMrMCnlv2nVrPQST5pHXUmdVAzYSVOvZeWPPg4CaV3WgEm7VOJ04qy0QKggK7eWSKpnCvDKlxkK9U2mB9Nmd4qipHuDu25t27CE9dAHX1gI-ebtniiz/s200/programming-an-hs1969-images.021.png" /></a></div>
<h3>The Four Stages of Creativity</h3>
<p>Preparation means to learn about something, to prime your brain. Then do
something else. Here are two ways to do this. 1) Study until you get stuck,
until you reach an impasse. 2) Study a little every day to keep the subject
percolating in your mind.</p>
<p>Incubation starts when we walk away from the problem. It is when our
unconscious mind takes over and keeps on working. Some good ways to let go of a
problem is to exercise, walk the dog, sleep, relax!</p>
<p>Illumination is the moment of insight.</p>
<p>Verification is optional for many types of insight, but when doing science it
is essential. It is when you prove that the insight is consistent with reality.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjyBZaQHTcptpZDy1Zq5oubp-exLNxv_vPRLt2Q0Hrr0UWNvZAEwk_2KnuvJlN2Qyt6f7YSuSWRe_67pj0HtETJdFz7Ng2HG7hK38J-yOT20zBwlI1BWdUUukLMg_4h6Rrtf-Zk/s1600/programming-an-hs1969-images.024.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjyBZaQHTcptpZDy1Zq5oubp-exLNxv_vPRLt2Q0Hrr0UWNvZAEwk_2KnuvJlN2Qyt6f7YSuSWRe_67pj0HtETJdFz7Ng2HG7hK38J-yOT20zBwlI1BWdUUukLMg_4h6Rrtf-Zk/s200/programming-an-hs1969-images.024.png" /></a></div>
<h3>Your Memory Bank</h3>
<pre><code>It is what you have in your memory bank, what you can
recall instantly that is important. If you have to look it up,
it is useless for creative thinking!
-- Linus Pauling
</code></pre>
<p>You cannot have an insight about things that you don't know anything about. If
you don't have the raw-material, you have nothing to work with. You cannot
Google for insights. Learn everything you can about a subject and then let go,
relax!</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhRI5BvU7I-YTtovpK5ySzcOjIlFy2zBTjKyqSexezfF-woMpiCC05O1C6hszl5Jvn0ZwQjIB2ZnHKoc89Sm3bkkC2-BEJw3_8si2NVdbi4ao2-e5LAbrjmSXvnSckggPq8OAF6/s1600/programming-an-hs1969-images.026.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhRI5BvU7I-YTtovpK5ySzcOjIlFy2zBTjKyqSexezfF-woMpiCC05O1C6hszl5Jvn0ZwQjIB2ZnHKoc89Sm3bkkC2-BEJw3_8si2NVdbi4ao2-e5LAbrjmSXvnSckggPq8OAF6/s200/programming-an-hs1969-images.026.png" /></a></div>
<h2>Deep Learning</h2>
<p>Deep learning is more than just knowing things. It is about understanding
things. Insights are crucial for this but so is knowing a lot of things.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGFVO8Mf8eBq36_AJ8d9z9dBTT_CfA5eUphONSeqBMmf3gZGVVOWzy_jqtpHUPgIdNQiPUp-xWek5FD7TUvDehwIun8Gl6dh32g-rLIIup9Wb5qkcdXiTmWyrM2c120lWl8yyu/s1600/programming-an-hs1969-images.027.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGFVO8Mf8eBq36_AJ8d9z9dBTT_CfA5eUphONSeqBMmf3gZGVVOWzy_jqtpHUPgIdNQiPUp-xWek5FD7TUvDehwIun8Gl6dh32g-rLIIup9Wb5qkcdXiTmWyrM2c120lWl8yyu/s200/programming-an-hs1969-images.027.png" /></a></div>
<h3>Working Memory and Long-Term Memory</h3>
<p>Working memory, also known as short-term memory is like a blackboard. It is
very limited and things have to be erased before we can put something else on
it.</p>
<p>Long-term memory, on the other hand, is surprisingly large. We don't know if
it has any limit at all.</p>
<p>Learning is essentially to move stuff from working memory into long-term
memory.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj7R6YOQMGIx_WZH52wvXhGaBkQPCZT0NvUkQNZOjDwrNLmmEJTHb7oA9Ip9l9hIC34_gQvakU6PuYcKiZxsiIjAVHcUEOuJGAdo2PbeM5qWqVX-wWZUq743lNIeiJCiq56lCeg/s1600/programming-an-hs1969-images.028.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj7R6YOQMGIx_WZH52wvXhGaBkQPCZT0NvUkQNZOjDwrNLmmEJTHb7oA9Ip9l9hIC34_gQvakU6PuYcKiZxsiIjAVHcUEOuJGAdo2PbeM5qWqVX-wWZUq743lNIeiJCiq56lCeg/s200/programming-an-hs1969-images.028.png" /></a></div>
<ul>
<li><strong>Encoding</strong> change the input from our senses into a format the brain can store.</li>
<li><strong>Consolidation</strong> recoding the memory to fit with other things that we know. This
is done largely unconsciously.</li>
<li><strong>Re-consolidation</strong> is when we recall a memory by reflecting or actually
retrieving it. This re-creates the memory and also changes it.</li>
</ul>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg2WOj7VgXyhBawvEeTY3gLC1Xx2irNpD4CGVX_14DlMF4r4X1YWMcA7H3yeCIjIY6Jt4xz2GzlvWLkpEEN5yUOTmSyOcHhyL-aSaK7PcOiaTQ0DGv0HDTmfd6TXLX918q56NVU/s1600/programming-an-hs1969-images.029.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg2WOj7VgXyhBawvEeTY3gLC1Xx2irNpD4CGVX_14DlMF4r4X1YWMcA7H3yeCIjIY6Jt4xz2GzlvWLkpEEN5yUOTmSyOcHhyL-aSaK7PcOiaTQ0DGv0HDTmfd6TXLX918q56NVU/s200/programming-an-hs1969-images.029.png" /></a></div>
<h3>Focused Mode and Diffused Mode</h3>
<p>Our brain works in two modes, focused mode and diffused mode.
<strong>Focused mode</strong> allows us to learn in a sequential step-by-step way. This is the
mode we need when we are encoding and learning new things.
<strong>Diffused mode</strong> is when we are not focusing on the problem, when our brain is
consolidating. This is when we have insights.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhGbMym_kzrzZHp5stT7xxoalBBeTMSXDc-4Qk2mNCWGSSRz5wQBzmiYJKsgp_BRsAxQ6mlwkFCuXtKqFlVqCAYhgDLUyJdl2m2lSnByyE7FP8FBzvUEvnsKztZT4HFBlY2YDHp/s1600/programming-an-hs1969-images.030.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhGbMym_kzrzZHp5stT7xxoalBBeTMSXDc-4Qk2mNCWGSSRz5wQBzmiYJKsgp_BRsAxQ6mlwkFCuXtKqFlVqCAYhgDLUyJdl2m2lSnByyE7FP8FBzvUEvnsKztZT4HFBlY2YDHp/s200/programming-an-hs1969-images.030.png" /></a></div>
<h3>Chunk</h3>
<p>A chunk is a piece of information related to prior knowledge. To learn
something is to create a new chunk. The more we learn about something the
bigger the chunk gets and the easier it is for us to relate it to other chunks.
A chunk can be kept in working memory as a single piece, without us having to
bring in all the details about it.</p>
<h3>Steps to Form a Chunk</h3>
<ol>
<li>Focus your <strong>undivided attention</strong> on the task.</li>
<li><strong>Understand</strong>, figure out what is the main idea and relate it to what you
know.</li>
<li><strong>Test</strong> yourself to <strong>verify</strong> that you understand.</li>
<li>Gain context by figuring out not only how but when to use it.</li>
</ol>
<h3>Relate Chunks</h3>
<p>To gain better understanding try to relate similar chunks together. A good way
to do this is by using metaphors and similes.
A simile is when we say something is like something else: <strong>Strong as an ox.</strong>
A metaphor is when we say something is something else: <strong>You are my sunshine.</strong></p>
<pre><code>Metaphors transforms the strange into the familiar.
--Twyla Tharp, The Creative Habit
</code></pre>
<h3>The Theory of Disuse (Forget-to-Learn)</h3>
<p>Any chunk of memory has two characteristics, storage strength and retrieval
strength. Storage strength is how well the item is learned. Retrieval strength
is how accessible the memory is at the time.</p>
<p>Storage strength increases monotonically, while retrieval strength varies with
context.</p>
<p>Both retrieval strength and storage strength increases with use.</p>
<p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhzX1mQlV17MBU80WhPTgo2jWHDPOGJ03i3mshrOo9JF6cYadGFWubteKILAbJcRzwfBDHYwnom2pnRGB1MpVng61gdSkZwrlqA3VJCvnV4e6kCfFZAR1tUaIKzTu1gfUpvbr78/s1600/programming-an-hs1969-images.035.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhzX1mQlV17MBU80WhPTgo2jWHDPOGJ03i3mshrOo9JF6cYadGFWubteKILAbJcRzwfBDHYwnom2pnRGB1MpVng61gdSkZwrlqA3VJCvnV4e6kCfFZAR1tUaIKzTu1gfUpvbr78/s200/programming-an-hs1969-images.035.png" /></a></div>
<strong>Retrieval</strong> is key. Retrieving causes us to re-learn the memory. And, the
harder it is to retrieve the deeper it gets stored in memory. More pain, more
gain!</p>
<p>If we see the mind as a forest, a newly learned memory is somewhere in the
forest. If we go to retrieve the memory, a path will be created to it. The more
you retrieve it the wider the path gets. If we instead look it up, the path
doesn't get created.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiCvcXubtYiZGjzZCXfQIPYkj-A9W11LkWOfL9Y2vi07g06WWCKhw0eFdcS2RbOwdWR_lbith_WQ6YHcezNXOjenKTtdWSp_Lnm7Fq7fwt6XCfS6g3z8k94J8IbLQPh4Dloqk8Q/s1600/programming-an-hs1969-images.037.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiCvcXubtYiZGjzZCXfQIPYkj-A9W11LkWOfL9Y2vi07g06WWCKhw0eFdcS2RbOwdWR_lbith_WQ6YHcezNXOjenKTtdWSp_Lnm7Fq7fwt6XCfS6g3z8k94J8IbLQPh4Dloqk8Q/s200/programming-an-hs1969-images.037.png" /></a></div>
<h3>Memory Strength Matrix</h3>
<p>In this 8 minutes long video, Destin from <a href="http://smartereveryday.com">http://smartereveryday.com</a>
goes through all the variations of a memory can have.</p>
<ul>
<li>When he starts, he knows how to ride a normal bike well. <strong>Strong storage and
retrieval strength</strong>.</li>
<li>After a bit of trying, he learns how it works to ride a reversed bike. <strong>Weak
storage and retrieval strength</strong>.</li>
<li>He manages to ride the reversed bike but falls as soon as he is distracted.
<strong>Strong retrieval and weak storage</strong>.</li>
<li>When he then tries to ride a normal bike again, ha cannot do it. Even though
riding a normal bike is <strong>strongly stored</strong>, it has <strong>weak retrieval strength</strong>.</li>
</ul>
<iframe width="640" height="360" src="https://www.youtube.com/embed/MFzDaBzBlL0?rel=0" frameborder="0" allowfullscreen></iframe>
<h3>Spaced Repetition</h3>
<p>Repeated retrieval is the mother of all learning, but it not good to just
repeat over and over without a pause. Because then we are leaning too much on
working memory. Pausing between repetitions is good for two reasons:</p>
<ol>
<li>It gives the diffuse mode time to work.</li>
<li>It lets us forget. Remember, we have to forget to learn.</li>
</ol>
<p>The most efficient way to learn something is to space it out into intervals.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigjLoiItqrAHFVW-EXWph64PuOaJi604GQD_mTzDduvjhA6LbpZXWH3dIG_aZWtu2kdrmBHzZJEZlMvWS5cFxmT2NL7KD0kGSX4QJX7LXuAl2Mgx4phHtMUXSBqWCV8rMAVA84/s1600/programming-an-hs1969-images.041.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigjLoiItqrAHFVW-EXWph64PuOaJi604GQD_mTzDduvjhA6LbpZXWH3dIG_aZWtu2kdrmBHzZJEZlMvWS5cFxmT2NL7KD0kGSX4QJX7LXuAl2Mgx4phHtMUXSBqWCV8rMAVA84/s200/programming-an-hs1969-images.041.png" /></a></div>
<p>How long should we wait before we try to retrieve a memory? The longer we
wait, while not totally forgetting it, the better. The harder the memory is to
retrieve the deeper it gets stored.</p>
<p>So what is the optimal time span? Of course scientists have a solution for this. :)</p>
<p>
If you have a test and you have decided to study for it three times. The
optimal time for the second study session is as described in the table. The
last study session should always be the day before the test.</p>
<p>But, spacing out is not the only important thing, context is also important.</p>
<h3>Context</h3>
<p>Context, under what circumstances are you studying.</p>
<p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZhOK_EcEkEHVYW2LOC_MhZwcHNGuUSWWzMpQBWQUjlk7RHkYFJEHda-X3WYbhaxStshzqrryo1PTxNSC1C0D8-aiiuGlyTYJxHsGAvhQs4sD7DMDwJV0dMY0KvgmsAWOCuPw1/s1600/programming-an-hs1969-images.043.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZhOK_EcEkEHVYW2LOC_MhZwcHNGuUSWWzMpQBWQUjlk7RHkYFJEHda-X3WYbhaxStshzqrryo1PTxNSC1C0D8-aiiuGlyTYJxHsGAvhQs4sD7DMDwJV0dMY0KvgmsAWOCuPw1/s200/programming-an-hs1969-images.043.png" /></a></div>
Location matters when you are studying. In one experiment, researchers let two
groups of people study vocabulary while under water. One group had to take the
test under water and the other group had to take the test on land. The group
who was submerged got better results than the group who was on land.</p>
<p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgCmsNK96K8Vi2mu7T7DX4kjav85dMXChNCajqy7ii0b3HBPe0GOYdy4cnZL_Bkq4mW6slG2RhAgeyMD1GAOpYgEX387WMX3opOts4lEdkDcpvVMsNejnbC7SMHiqHpmaz2Ag8r/s1600/programming-an-hs1969-images.044.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgCmsNK96K8Vi2mu7T7DX4kjav85dMXChNCajqy7ii0b3HBPe0GOYdy4cnZL_Bkq4mW6slG2RhAgeyMD1GAOpYgEX387WMX3opOts4lEdkDcpvVMsNejnbC7SMHiqHpmaz2Ag8r/s200/programming-an-hs1969-images.044.png" /></a></div>
In another study, the subjects were studying after smoking marijuana. Same
result there, the group who were stoned got better results than the unstoned
group.</p>
<p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiUvbt7xlVXvSmGOJ8MraS39EI-2ORZww3vn-NGckru0Ddq_azJw1ajFgJItvuE_uHy9yzHN-WkPMmXNDsCDcAoL7Q68RKPcvZ0LidCiRi2opWDHgPag2y6l6JlXwTdMz9g4HYO/s1600/programming-an-hs1969-images.045.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiUvbt7xlVXvSmGOJ8MraS39EI-2ORZww3vn-NGckru0Ddq_azJw1ajFgJItvuE_uHy9yzHN-WkPMmXNDsCDcAoL7Q68RKPcvZ0LidCiRi2opWDHgPag2y6l6JlXwTdMz9g4HYO/s200/programming-an-hs1969-images.045.png" /></a></div>
And the same thing again with music. If students were allowed to listen to the
same music that they had used while studying they got better results. It does
not matter what kind of music it is, Mozart is not better than AC/DC.
Interestingly not listening to music didn't have this effect. Not listening to
music does not seem to provide a context.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi4xCX98pk3gGWuAy2V5RMPzP6xJ9GLOM6N3EgvNPKLpIy_TR-EoSck1bCNyloq4IDFjSkCfHx2uOg8p5R-l4-g3FnQR8xPIIHuKXNu6Ejgi_b_oSip2mp1CWU_twxyWOf7DiHz/s1600/programming-an-hs1969-images.046.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi4xCX98pk3gGWuAy2V5RMPzP6xJ9GLOM6N3EgvNPKLpIy_TR-EoSck1bCNyloq4IDFjSkCfHx2uOg8p5R-l4-g3FnQR8xPIIHuKXNu6Ejgi_b_oSip2mp1CWU_twxyWOf7DiHz/s200/programming-an-hs1969-images.046.png" /></a></div>
<h3>Variation</h3>
<p>You may be thinking, "How in the world am I going to get my teacher to allow me
to take my tests stoned, under water, while listening to AC/DC?". This is not
the point, the point is variation. If we learn under varying conditions our
brain gets better at retrieving the information under conditions where we
haven't practiced. And you can vary anything:</p>
<ul>
<li>The color of the paper you are writing on.</li>
<li>Your mood, are you happy or sad?</li>
<li>Inside or outside</li>
<li>Morning or evening</li>
<li>Sitting still or exercising</li>
<li>etc.</li>
</ul>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh40vxo0olhdds8uNqfrfpPh72DIp57bpFGWicCfBp8f4QR3N_HnxD3r-vg7d5QiE9Bqz3m0j15UvBbjOrzwVytnznW1HnhRWdjBTTBk4H889MEacGxgJuakW6_MMPrTZeueNmS/s1600/programming-an-hs1969-images.047.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh40vxo0olhdds8uNqfrfpPh72DIp57bpFGWicCfBp8f4QR3N_HnxD3r-vg7d5QiE9Bqz3m0j15UvBbjOrzwVytnznW1HnhRWdjBTTBk4H889MEacGxgJuakW6_MMPrTZeueNmS/s200/programming-an-hs1969-images.047.png" /></a></div>
<h2>Interleaving</h2>
<p>Studying the same thing over and over again is called massed practice. It is
commonly believed to be good, because it feels like you are learning fast, but
it it's an illusion. It is much more effective to interleave your practice even
though it doesn't feel that way.</p>
<p>Don't study the same thing over and over, vary the tasks. Don't just learn to
calculate the area of a circle over and over, vary the practice with different
figures.</p>
<p>In one study children where told to practice throwing bean bags. One group
practiced with a distance of three feet. Another group practiced from two and
four feet. The day after they had a competition, <strong>from three feet</strong>. The group
which had used interleaved practice won easily even though they had never
practiced on this distance.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgvgcGLyQ3OS8wVL92boWB6U5y2nIzzHDNLsZIEsSXYhh6nc4ClGWL76xG_Wc8Y8FwIKVr0VBK8DmRrzTnq6zBcU_xyI2C27UdGPBmhO2YFOgrj2YSmQ_hEBlIoUimmiHc6uRG3/s1600/programming-an-hs1969-images.050.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgvgcGLyQ3OS8wVL92boWB6U5y2nIzzHDNLsZIEsSXYhh6nc4ClGWL76xG_Wc8Y8FwIKVr0VBK8DmRrzTnq6zBcU_xyI2C27UdGPBmhO2YFOgrj2YSmQ_hEBlIoUimmiHc6uRG3/s200/programming-an-hs1969-images.050.png" /></a></div>
<h2>Desirable Difficulties</h2>
<p>The harder it is for us to retrieve the information the better we learn. It is
better to attempt to retrieve and be wrong than to not attempt at all as long
as we get feedback about what the correct answer is.</p>
<p>Think of learning as exercising. The harder it is, the stronger we get. No
pain, no gain!</p>
<p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhgxAg-kUNbyrXi_gGQG80e9waeYrqmQy3s4Ca9_rHhiEP6AzqQ-IxMtv9QAn1AGOEAm2nh4xxJg3m9bLIUFO7dyolZZtdkDm1GyvyRrs-w-M6-55gtVab7BnQH6h39OYzqlQHP/s1600/programming-an-hs1969-images.051.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhgxAg-kUNbyrXi_gGQG80e9waeYrqmQy3s4Ca9_rHhiEP6AzqQ-IxMtv9QAn1AGOEAm2nh4xxJg3m9bLIUFO7dyolZZtdkDm1GyvyRrs-w-M6-55gtVab7BnQH6h39OYzqlQHP/s200/programming-an-hs1969-images.051.png" /></a></div>
<h2>Testing</h2>
<p>Testing is a great way to learn mainly for two reasons.</p>
<ol>
<li>It tells you what you know and don't know.</li>
<li>It is an extreme form of retrieval. It focuses our mind, because when we
really want to retrieve the information we try extra hard.</li>
</ol>
<p>It is even helpful to pre-test what you haven't learned yet. This tells your
brain that you are interested in this information and makes it more receptive
and focused when you study.</p>
<p>Test come in varying degrees of difficulty.</p>
<p>From easy to hard:</p>
<ul>
<li>Multiple choice</li>
<li>Fill in the blank</li>
<li>Reply with a sentence</li>
<li>Write an essay</li>
</ul>
<h3>Classroom Testing</h3>
<p>Multiple studies have shown that students in classes that have many tests, one
per week gets better grades than students who only get two tests per semester.
A full grade better on average!</p>
<h3>Self-Testing</h3>
<p>It is not necessary to have formal tests. We can verify that we know by asking
ourselves <strong>and answering</strong> questions.</p>
<ul>
<li>Do I understand what this means?</li>
<li>What are the basic ideas in this text?</li>
<li>Can I explain this to someone else?</li>
<li>How does it relate to what I already know?</li>
</ul>
<h3>Sleep</h3>
<p>Sleep is very good for our brain primarily for two reasons. First, being awake
creates toxic products in our brain and sleeping cleans them out. Second, while
we sleep our brain tidies up and removes unimportant ideas while simultaneously
strengthening ideas that it believes are important to you.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhFRx0st2c1J0PfYIc8khCsa2yA7TeO8jB579iGoQ74aHtNOJiNckn96JSbzqVNPbisVIYOm_zdfCvGqhhRrb6f99OQY_QBYKSNTNkb1cEhFzdEDvpph4TQJ5M8F6-2rKlGWCDZ/s1600/programming-an-hs1969-images.058.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhFRx0st2c1J0PfYIc8khCsa2yA7TeO8jB579iGoQ74aHtNOJiNckn96JSbzqVNPbisVIYOm_zdfCvGqhhRrb6f99OQY_QBYKSNTNkb1cEhFzdEDvpph4TQJ5M8F6-2rKlGWCDZ/s200/programming-an-hs1969-images.058.png" /></a></div>
<h2>Shallow Learning or Perceptual Learning</h2>
<p>Shallow learning or perceptual learning is learning without understanding. It
is pattern recognition and it is great for categorization. Perceptual learning
is active, our eyes or other senses are searching for the right clues
automatically and tunes itself. Here are some examples.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEikeFdqrEk1y9I70BmxRsdjKOFh2zyspftPEQEbdUTizrYuYA7mq-0EI2DiqKwKjXSkE15i7vSxf3dTJ9rbSnuwiCSUlx0f5JNCiIpDZV_wv27akPgHqq68VdaGVvT3gnFzD6Xr/s1600/programming-an-hs1969-images.059.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEikeFdqrEk1y9I70BmxRsdjKOFh2zyspftPEQEbdUTizrYuYA7mq-0EI2DiqKwKjXSkE15i7vSxf3dTJ9rbSnuwiCSUlx0f5JNCiIpDZV_wv27akPgHqq68VdaGVvT3gnFzD6Xr/s200/programming-an-hs1969-images.059.png" /></a></div>
<h3>Chicken Sexing</h3>
<p>To find out what sex a chicken has is very difficult and the experts who can do
it cannot describe how they do it in a way that is understandable to an
outsider. So how do you learn? By example. Make a wild guess. After each
guess, the master chick-sexer gives you feedback.Yes, no, no, yes. And,
eventually, you just start to make correct guesses without knowing how.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1U-0YNUR_ed3nhiIRMvCRtU6oZI_XKJH5EUQWLXL1mR3SiGE4sa53y6kOGAQ3D9ZoXew_t0-mW6MUb2ndyIhHAO_WU-VJXTxwLcy2m5-gyKx_Ww2j0b8cb0jGoOv5fdM9ssyn/s1600/programming-an-hs1969-images.060.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1U-0YNUR_ed3nhiIRMvCRtU6oZI_XKJH5EUQWLXL1mR3SiGE4sa53y6kOGAQ3D9ZoXew_t0-mW6MUb2ndyIhHAO_WU-VJXTxwLcy2m5-gyKx_Ww2j0b8cb0jGoOv5fdM9ssyn/s200/programming-an-hs1969-images.060.png" /></a></div>
<h3>Flight Training</h3>
<p>When a pilot learns to fly by instruments he has six instruments he needs to know.
Airspeed Indicator, Attitude Indicator, Altimeter, Vertical Speed Indicator,
Heading Indicator, Turn Coordinator.</p>
<p>When novices try to fly by instruments, they have a hard time understanding
what is going on. While they focus on one instrument the other instruments
change and it is difficult to get an understanding of what it means. Experts
pilots, on the other hand, quickly glance at the instruments and instantly know
what is going on.</p>
<p>By simple flash-card training a novice can get as good at instrument reading in
<strong>one</strong> hour that normally takes <strong>a thousand</strong> hours. This training takes
advantage of the perceptual learning ability our brain has for quickly
categorizing the information.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQRptAYf5LMVYo8TWLt7-0sfUDjCsaaTnSsd8-nIBvBQDXznpbxqS0T9sc8iEuTw3MmqZJfBJDAbdRCdQPVksYJujmPb0J_uAebSs3tGJa_SEVF3ds9ewlu-4X2lBAxRRI0Zch/s1600/programming-an-hs1969-images.061.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQRptAYf5LMVYo8TWLt7-0sfUDjCsaaTnSsd8-nIBvBQDXznpbxqS0T9sc8iEuTw3MmqZJfBJDAbdRCdQPVksYJujmPb0J_uAebSs3tGJa_SEVF3ds9ewlu-4X2lBAxRRI0Zch/s200/programming-an-hs1969-images.061.png" /></a></div>
<h3>Art Recognition</h3>
<p>Flash-cards can also be used to learn to categorize painting styles. Without
having any other knowledge of the paintings or the painters we can learn to
categorize a painting as surrealism, minimalism, or any other style by just
practicing over and over.</p>
<h2>Obstacles</h2>
<p>The biggest obstacle to our learning is ourselves. We have a tendency to do
what feels good and not what is good and it is easy to fool ourselves to think
that we know something when we actually don't.</p>
<p>Rickard Feynman once said:</p>
<pre><code>The first principle is that you must not fool yourself - and you are the
easiest person to fool!
</code></pre>
<p>So how do we fool ourselves?</p>
<h3>Illusions of Understanding</h3>
<h4>Re-reading</h4>
<p>Reading a text over and over gives us the illusion that we know the text since
we become familiar with it. We recognize the text as we read it. This is not
the same as knowing it. <strong>The solution is retrieval practice!</strong></p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiw-q7G9g9bSjXPiBrNX21-lxZtGoJngWSxzPC_Q4MCYCfVBAoNeoFa7ShWNW7psMDWybkJ4QJUmF6IuAK811FbxMiCszitbj0FBPUV0OHFPPRdehFLqSSa1DPni8EWJ4XPM4YQ/s1600/programming-an-hs1969-images.066.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiw-q7G9g9bSjXPiBrNX21-lxZtGoJngWSxzPC_Q4MCYCfVBAoNeoFa7ShWNW7psMDWybkJ4QJUmF6IuAK811FbxMiCszitbj0FBPUV0OHFPPRdehFLqSSa1DPni8EWJ4XPM4YQ/s200/programming-an-hs1969-images.066.png" /></a></div>
<h4>Highlighting</h4>
<p>Highlighting text is another way we fool ourselves. If we look at a page which
is highlighted all over, it is easy to think that we already know it. <strong>The
solution is retrieval practice!</strong></p>
<h4>Looking at the Answer</h4>
<p>If we are solving problems, it is very easy to look at the answer before we
have actually tried to solve the problem. We look at the answer and we tell
ourselves that we could have solved this without looking. <strong>The solution in this
case is to actually try to solve the problem</strong>. Even if we fail to solve it, we
learn better than by just looking at the answer.</p>
<h4>Googling</h4>
<p>By Googling instead of trying to retrieve a memory we rob our brain of the
extra storage and retrieval strength that come with retrieval. It is better not
to remember and let our diffuse mode go to work and let it pop into our head
while we least expect it.</p>
<h4>Echoing Other People's Words</h4>
<p>If you know a text verbatim, in exactly the same words used by someone else,
odds are that you don't really know what you are saying. If you cannot explain
it in your own words, you are probably fooling yourself.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJcILG2mp816mJuyo9_Hw92rRQyB-H0zC1zHu4FdzIJR9dxLpgBc3eivBuHsf_BQHQIAFgAOwh2gOlbh9kvcb_E2apxYlwLojPt1h3LI04IM834S1m-sf4jDTGTZ5PIzig57WI/s1600/programming-an-hs1969-images.070.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJcILG2mp816mJuyo9_Hw92rRQyB-H0zC1zHu4FdzIJR9dxLpgBc3eivBuHsf_BQHQIAFgAOwh2gOlbh9kvcb_E2apxYlwLojPt1h3LI04IM834S1m-sf4jDTGTZ5PIzig57WI/s200/programming-an-hs1969-images.070.png" /></a></div>
<h3>Distractions</h3>
<p>Another big obstacle to learning is distractions. The brain needs focus time in
order to store memories deep enough for later retrieval.</p>
<p>Studies have shown that students who learn while watching TV at the same time
may actually know it quite well immediately after studying. But, when tested
later their retrieval ability was abysmal.</p>
<p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEghgjKkqQUI7Dhj9EgvuNripiUzuCfKXu1FH8iQUTBbOqT5q3s5RQVDhSXL0gkIkdrmKT4dKqU5_PsgrB3MiaNOrQIioMx9Me0V_OGxgL2qPYzrJ5xOkmHtcLSSjWKXiVirkU9t/s1600/programming-an-hs1969-images.071.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEghgjKkqQUI7Dhj9EgvuNripiUzuCfKXu1FH8iQUTBbOqT5q3s5RQVDhSXL0gkIkdrmKT4dKqU5_PsgrB3MiaNOrQIioMx9Me0V_OGxgL2qPYzrJ5xOkmHtcLSSjWKXiVirkU9t/s200/programming-an-hs1969-images.071.png" /></a></div>
If you have problems focusing while studying, the pomodoro technique may help
you. Set a timer for an amount of time, 20 minutes or half an hour, and vow
not to be extracted until the timer rings.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgG_MVSeLnb8yeRrN0Oe2gikxyOilWiMHTVAPyT5kwuOK4D0-wfW80h_UjyJr2zlzJX0UaclnAUiutSF1YI5HbPlgRszvNqfvnBoRo6RHAGrucmLhZeuJOChKRuFeIRDXN6GquR/s1600/programming-an-hs1969-images.072.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgG_MVSeLnb8yeRrN0Oe2gikxyOilWiMHTVAPyT5kwuOK4D0-wfW80h_UjyJr2zlzJX0UaclnAUiutSF1YI5HbPlgRszvNqfvnBoRo6RHAGrucmLhZeuJOChKRuFeIRDXN6GquR/s200/programming-an-hs1969-images.072.png" /></a></div>
<h3>Procrastination</h3>
<p>Procrastination is the avoidance of doing a task which needs to be
accomplished. It is the practice of doing more pleasurable things in place of
less pleasurable ones, or carrying out less important tasks instead of more
important ones, thus putting off impending tasks to a later time.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEim7__Fik3OJXWcMEAslV0QtnZ0scQu04GdQBYqbKJql7FXpkdtJ0VNfob0fJjApx6ixud4ptjptCNfeDfWb1AWv8xjFo-SPBFGA5ya3kkCcDFgvxu9yCmuS0w3lRSks-IYgyUu/s1600/programming-an-hs1969-images.073.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEim7__Fik3OJXWcMEAslV0QtnZ0scQu04GdQBYqbKJql7FXpkdtJ0VNfob0fJjApx6ixud4ptjptCNfeDfWb1AWv8xjFo-SPBFGA5ya3kkCcDFgvxu9yCmuS0w3lRSks-IYgyUu/s200/programming-an-hs1969-images.073.png" /></a></div>
<h4>Start</h4>
<p>The key to overcome procrastination is to Just Start. Very often the
anticipation of doing something is worse than actually doing it.</p>
<p>The writer Dorothy Parker once said:</p>
<pre><code>Writing is the art of applying the ass to the seat.
</code></pre>
<h3>Solutions to Insight Problems</h3>
<p>What historical person does <strong>"Horobod"</strong> symbolize? <strong>Robin Hood</strong></p>
<p>A window washer fell from a 40-foot ladder without hurting himself. How is this
possible? <strong>He fell from the bottom step.</strong></p>
<p>Thiss sentence has thre errors? What are they? <strong>Two spelling errors, and the
semantic error that the sentence only has two errors while claiming to have
three.</strong></p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjvop1GQFpj7fOmGOLNCCgjwN9wYqoLK9f90zO-zvM7PqJ2KDW1eCiuRwUksyCFyTcYFPwoP8Q8OdcSzbXu-Htm6yvCa0SF8Dn1Wx_m_SwO1gl0cfCDvd3MZQc9IarHHIz0UtGX/s1600/programming-an-hs1969-images.077.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjvop1GQFpj7fOmGOLNCCgjwN9wYqoLK9f90zO-zvM7PqJ2KDW1eCiuRwUksyCFyTcYFPwoP8Q8OdcSzbXu-Htm6yvCa0SF8Dn1Wx_m_SwO1gl0cfCDvd3MZQc9IarHHIz0UtGX/s200/programming-an-hs1969-images.077.png" /></a></div>
<h4>Pyramid solution</h4>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi536bTEC1Pu9nPlwTvUNscmOZ-Eljvq0hD32GaIQUQsbXK3zP1aF3G-7BM_3-0O0gpYThRi5qK4pKnx2fcS-yfDuKYoyBCpAoJFjg0lYgUmJteVFbZ59cr7-3woLYM7FaxZbl6/s1600/programming-an-hs1969-images.079.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi536bTEC1Pu9nPlwTvUNscmOZ-Eljvq0hD32GaIQUQsbXK3zP1aF3G-7BM_3-0O0gpYThRi5qK4pKnx2fcS-yfDuKYoyBCpAoJFjg0lYgUmJteVFbZ59cr7-3woLYM7FaxZbl6/s200/programming-an-hs1969-images.079.png" /></a></div>
<h4>9-dots solution</h4>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi_BsXvR_Sldui9mLYRUbKpjLgGETgfRltTJQoIdajZgBIqgc7S059L619bC8LcusR2l8YG3bzw4BzJqin3FVNQFrStCAV0eRtp4T6mFBEfxvP5U186zpnMVUJw1usemP309ouY/s1600/programming-an-hs1969-images.080.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi_BsXvR_Sldui9mLYRUbKpjLgGETgfRltTJQoIdajZgBIqgc7S059L619bC8LcusR2l8YG3bzw4BzJqin3FVNQFrStCAV0eRtp4T6mFBEfxvP5U186zpnMVUJw1usemP309ouY/s200/programming-an-hs1969-images.080.png" /></a></div>
<h3>Happiness</h3>
<p>Martin Seligman is the author of the book, Authentic Happiness. He describes
how, in a study on creative problem solving, subjects who were put in a good
mood produced much better results. Happy people perform better! Increasing
happiness, increases the likelihood of insight.</p>
<p>So here is story to put a smile on your face and to increase your creativity.</p>
<pre><code>
A priest fell into a well, but managed to get a hold of a
small vine before falling into the abyss. When he had been
hanging there for almost an hour, he heard a loud thunder and a voice from above.
"This is your God speaking, if you let go of the vine I will save you!"
The priest contemplated this for a while and then he yelled, "Is there anybody
else up there?"
</code></pre>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjUuv5SK0q22UgTqTuUlcZFknThrIheh3z6PDFDp_nT_hVD1fGGofCqxi043SVmzUqySAC9JbO35ktTZuuwau5gTEkGKi7QmIPHX7shpjPegkYVIYg3V3xjU1fZYFbsLM04AtEO/s1600/programming-an-hs1969-images.082.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjUuv5SK0q22UgTqTuUlcZFknThrIheh3z6PDFDp_nT_hVD1fGGofCqxi043SVmzUqySAC9JbO35ktTZuuwau5gTEkGKi7QmIPHX7shpjPegkYVIYg3V3xjU1fZYFbsLM04AtEO/s200/programming-an-hs1969-images.082.png" /></a></div>
<h2>Summary</h2>
<p>Insights help us to understand and relate things. In order to have more
insights we have to learn things deeply and then let go. The best way to learn
new things is by retrieval practice. Our biggest obstacle to learning is that
we fool ourselves that we know things we don't know. Happy people have more
insights, so by seeing the light side of things we not only feel better, we
perform better.</p>
<h2>References</h2>
<ul>
<li><a href="http://www.amazon.com/Mind-Numbers-Science-Flunked-Algebra/dp/039916524X?tag=thtasta-20">A Mind for Numbers</a></li>
<li><a href="http://www.amazon.com/Aha-Moments-Insight-Shape-World/dp/0199338876?tag=thtasta-20">Aha!: The Moments of Insight that Shape Our World</a></li>
<li><a href="http://www.amazon.com/Authentic-Happiness-Psychology-Potential-Fulfillment/dp/0743222989?tag=thtasta-20">Authentic Happiness: Using the New Positive Psychology to Realize Your Potential for Lasting Fulfillment</a></li>
<li><a href="http://www.amazon.com/How-We-Learn-Surprising-Happens/dp/0812984293?tag=thtasta-20">How We Learn: The Surprising Truth About When, Where, and Why It Happens</a></li>
<li><a href="http://www.amazon.com/Make-Stick-Science-Successful-Learning/dp/0674729013?tag=thtasta-20">Make It Stick: The Science of Successful Learning</a></li>
<li><a href="http://www.amazon.com/Moonwalking-Einstein-Science-Remembering-Everything/dp/0143120530?tag=thtasta-20">Moonwalking with Einstein: The Art and Science of Remembering Everything</a></li>
<li><a href="http://www.amazon.com/5-Elements-Effective-Thinking/dp/0691156662?tag=thtasta-20">The 5 Elements of Effective Thinking</a></li>
<li><a href="http://www.amazon.com/The-Creative-Habit-Learn-Life/dp/1480589837?tag=thtasta-20">The Creative Habit</a></li>
<li><a href="http://www.amazon.com/The-War-Art-Winning-Creative/dp/1501260626?tag=thtasta-20">The War of Art</a></li>
<li><a href="http://www.amazon.com/Thinking-Fast-Slow-Daniel-Kahneman/dp/0374533555?tag=thtasta-20">Thinking, Fast and Slow</a></li>
</ul>Anders Janmyrhttp://www.blogger.com/profile/16081537018884349622noreply@blogger.com2tag:blogger.com,1999:blog-34049130.post-84939141578627510622015-11-23T10:49:00.001+01:002015-11-23T10:49:43.606+01:00Simple Clustering with Docker Swarm and Nginx
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEisf0EtNVFxpbeYdvOe-vLSQv0S_uRVCCgNl_J-oKrqyDpq9bmejlmHhQq3kRy11eCDsj1rwwu5Zb7RyIchfW9VG1wjPj6hNBvz-YKQWeMJga8wVyzgYmVmk3ZFDG_aw-2zn7JS/s1600/docker-swarm.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEisf0EtNVFxpbeYdvOe-vLSQv0S_uRVCCgNl_J-oKrqyDpq9bmejlmHhQq3kRy11eCDsj1rwwu5Zb7RyIchfW9VG1wjPj6hNBvz-YKQWeMJga8wVyzgYmVmk3ZFDG_aw-2zn7JS/s320/docker-swarm.png" /></a></div>
<p>Bringing up your own cluster has never been easier. The <a href="https://blog.docker.com/2015/11/swarm-1-0/">recent 1.0 release of
Docker Swarm</a> signals that the
Docker team feel that Swarm is ready for production.</p>
<p>I've been running a bunch of applications on Docker for a while now, but I have
managed the containers on the single machine level instead of as a cluster.
With the release of Swarm 1.0, I believe it is time to start clustering my
machines.</p>
<h2>Spinning Up the Swarm</h2>
<p>How to spin up a Swarm for development is described well in the <a href="http://docs.docker.com/swarm/install-w-machine/">Docker
documentation</a> and I'm not going
to describe it in depth here. I'll settle for the commands and extra
documentation when I feel that it may be called for.</p>
<p>I'm using the Swarm for development with VirtualBox here, but it is simple to
substitute any of the supported <a href="https://docs.docker.com/machine/drivers/">docker-machine
providers</a>.</p>
<h3>Create a Token</h3>
<p>Create a token with the Docker Hub discovery service. When running this in
production you should probably setup an <a href="http://docs.docker.com/swarm/discovery/">alternate discovery backend</a>
to avoid the external dependency.</p>
<blockquote><pre><tt><i><font color="#9A1900"># Create and save a token, using the Docker-Hub discovery service, default</font></i>
$ <font color="#009900">token</font><font color="#990000">=</font><font color="#009900">$(</font>docker run swarm create<font color="#990000">)</font>
</tt></pre></blockquote>
<h3>Create a Swarm Manager</h3>
<p>The swarm manager will be used to control the swarm. It should be protected
from access from anyone but you. I'll simulate this here by setting
<code>--engine-label public=no</code>. This is just a tag and you would have to make sure
that you setup the manager protected from public access. It is possible to use
multiple labels to tag the engine with all the qualities of this machine.</p>
<blockquote><pre><tt><i><font color="#9A1900"># Create a swarm manager using the token</font></i>
$ docker-machine create <font color="#990000">\</font>
-d virtualbox <font color="#990000">\</font>
--swarm <font color="#990000">\</font>
--swarm-master <font color="#990000">\</font>
--swarm-discovery token<font color="#990000">://</font><font color="#009900">$token</font> <font color="#990000">\</font>
--engine-label <font color="#009900">public</font><font color="#990000">=</font>no <font color="#990000">\</font>
swarm-master
</tt></pre></blockquote>
<h3>Create a Publicly Accessible Machine</h3>
<p>In this demo I'm only spinning up another VirtualBox machine and I'm giving it
the <code>--engine-label public=yes</code> to allow me to discover this box in the swarm.</p>
<blockquote><pre><tt><i><font color="#9A1900"># Create a new node named frontend and label it public</font></i>
$ docker-machine create <font color="#990000">\</font>
-d virtualbox <font color="#990000">\</font>
--swarm <font color="#990000">\</font>
--swarm-discovery token<font color="#990000">://</font><font color="#009900">$token</font> <font color="#990000">\</font>
--engine-label <font color="#009900">public</font><font color="#990000">=</font>yes <font color="#990000">\</font>
frontend
</tt></pre></blockquote>
<h3>Create a Couple of Additional Non-Public Machines</h3>
<p>Here I start a couple of machines with an additional <code>--engine-label</code>.
One with <code>model=high-memory</code> and one with <code>model=large-disk</code></p>
<blockquote><pre><tt><i><font color="#9A1900"># Create two more nodes named backend1 and backend2, with label public=no</font></i>
$ docker-machine create <font color="#990000">\</font>
-d virtualbox <font color="#990000">\</font>
--swarm <font color="#990000">\</font>
--swarm-discovery token<font color="#990000">://</font><font color="#009900">$token</font> <font color="#990000">\</font>
--engine-label <font color="#009900">public</font><font color="#990000">=</font>no <font color="#990000">\</font>
--engine-label <font color="#009900">model</font><font color="#990000">=</font>high-memory <font color="#990000">\</font>
backend1
$ docker-machine create <font color="#990000">\</font>
-d virtualbox <font color="#990000">\</font>
--swarm <font color="#990000">\</font>
--swarm-discovery token<font color="#990000">://</font><font color="#009900">$token</font> <font color="#990000">\</font>
--engine-label <font color="#009900">public</font><font color="#990000">=</font>no <font color="#990000">\</font>
--engine-label <font color="#009900">model</font><font color="#990000">=</font>large-disk <font color="#990000">\</font>
backend2
</tt></pre></blockquote>
<h3>List the Swarm</h3>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgh8LsyQVCtd4ku09OopZksqe781GwnZ-uJPCk_20cgR4WR2RJqEzOy-7A9WXHamhJ1b8tkH05eLpJjz8gWiOBn973pNF8VUk2G0rM8fwKAIvo_B-Oj1D2mkE1orf5JpknVPq6g/s1600/the-swarm-machines.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgh8LsyQVCtd4ku09OopZksqe781GwnZ-uJPCk_20cgR4WR2RJqEzOy-7A9WXHamhJ1b8tkH05eLpJjz8gWiOBn973pNF8VUk2G0rM8fwKAIvo_B-Oj1D2mkE1orf5JpknVPq6g/s1600/the-swarm-machines.png" /></a></div>
<blockquote><pre><tt><i><font color="#9A1900"># List your machines</font></i>
$ docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM
backend1 - virtualbox Running tcp<font color="#990000">://</font><font color="#993399">192.168</font><font color="#990000">.</font><font color="#993399">99.103</font><font color="#990000">:</font><font color="#993399">2376</font> swarm-master
backend2 - virtualbox Running tcp<font color="#990000">://</font><font color="#993399">192.168</font><font color="#990000">.</font><font color="#993399">99.104</font><font color="#990000">:</font><font color="#993399">2376</font> swarm-master
frontend - virtualbox Running tcp<font color="#990000">://</font><font color="#993399">192.168</font><font color="#990000">.</font><font color="#993399">99.102</font><font color="#990000">:</font><font color="#993399">2376</font> swarm-master
swarm-master - virtualbox Running tcp<font color="#990000">://</font><font color="#993399">192.168</font><font color="#990000">.</font><font color="#993399">99.101</font><font color="#990000">:</font><font color="#993399">2376</font> swarm-master <font color="#990000">(</font>master<font color="#990000">)</font>
</tt></pre></blockquote>
<h3>Connect to the Swarm</h3>
<p>Configure the docker client to connect to it.</p>
<blockquote><pre><tt><i><font color="#9A1900"># List the environment needed to connect to the swarm</font></i>
$ docker-machine env --swarm swarm-master
<b><font color="#0000FF">export</font></b> <font color="#009900">DOCKER_TLS_VERIFY</font><font color="#990000">=</font><font color="#FF0000">"1"</font>
<b><font color="#0000FF">export</font></b> <font color="#009900">DOCKER_HOST</font><font color="#990000">=</font><font color="#FF0000">"tcp://192.168.99.101:3376"</font>
<b><font color="#0000FF">export</font></b> <font color="#009900">DOCKER_CERT_PATH</font><font color="#990000">=</font><font color="#FF0000">"/Users/andersjanmyr/.docker/machine/machines/swarm-master"</font>
<b><font color="#0000FF">export</font></b> <font color="#009900">DOCKER_MACHINE_NAME</font><font color="#990000">=</font><font color="#FF0000">"swarm-master"</font>
<i><font color="#9A1900"># Run this command to configure your shell:</font></i>
<i><font color="#9A1900"># eval "$(docker-machine env --swarm swarm-master)"</font></i>
<i><font color="#9A1900"># Configure docker to use the swarm-master</font></i>
$ <b><font color="#0000FF">eval</font></b> <font color="#009900">$(</font>docker-machine env --swarm swarm-master<font color="#990000">)</font>
<i><font color="#9A1900"># List information about the cluster, output is trimmed</font></i>
$ docker info
Containers<font color="#990000">:</font> <font color="#993399">4</font>
Images<font color="#990000">:</font> <font color="#993399">4</font>
Role<font color="#990000">:</font> primary
Strategy<font color="#990000">:</font> spread
Filters<font color="#990000">:</font> health<font color="#990000">,</font> port<font color="#990000">,</font> dependency<font color="#990000">,</font> affinity<font color="#990000">,</font> constraint
Nodes<font color="#990000">:</font> <font color="#993399">4</font>
backend1<font color="#990000">:</font> <font color="#993399">192.168</font><font color="#990000">.</font><font color="#993399">99.103</font><font color="#990000">:</font><font color="#993399">2376</font>
Containers<font color="#990000">:</font> <font color="#993399">1</font>
Reserved CPUs<font color="#990000">:</font> <font color="#993399">0</font> <font color="#990000">/</font> <font color="#993399">1</font>
Reserved Memory<font color="#990000">:</font> <font color="#993399">0</font> B <font color="#990000">/</font> <font color="#993399">1.021</font> GiB
Labels<font color="#990000">:</font> <font color="#009900">model</font><font color="#990000">=</font>high-memory<font color="#990000">,</font> <font color="#009900">provider</font><font color="#990000">=</font>virtualbox<font color="#990000">,</font> <font color="#009900">public</font><font color="#990000">=</font>no<font color="#990000">,</font> <font color="#009900">storagedriver</font><font color="#990000">=</font>aufs
backend2<font color="#990000">:</font> <font color="#993399">192.168</font><font color="#990000">.</font><font color="#993399">99.104</font><font color="#990000">:</font><font color="#993399">2376</font>
Containers<font color="#990000">:</font> <font color="#993399">5</font>
Reserved CPUs<font color="#990000">:</font> <font color="#993399">0</font> <font color="#990000">/</font> <font color="#993399">1</font>
Reserved Memory<font color="#990000">:</font> <font color="#993399">0</font> B <font color="#990000">/</font> <font color="#993399">1.021</font> GiB
Labels<font color="#990000">:</font> <font color="#009900">model</font><font color="#990000">=</font>large-disk<font color="#990000">,</font> <font color="#009900">provider</font><font color="#990000">=</font>virtualbox<font color="#990000">,</font> <font color="#009900">public</font><font color="#990000">=</font>no<font color="#990000">,</font> <font color="#009900">storagedriver</font><font color="#990000">=</font>aufs
frontend<font color="#990000">:</font> <font color="#993399">192.168</font><font color="#990000">.</font><font color="#993399">99.102</font><font color="#990000">:</font><font color="#993399">2376</font>
Containers<font color="#990000">:</font> <font color="#993399">1</font>
Reserved CPUs<font color="#990000">:</font> <font color="#993399">0</font> <font color="#990000">/</font> <font color="#993399">1</font>
Reserved Memory<font color="#990000">:</font> <font color="#993399">0</font> B <font color="#990000">/</font> <font color="#993399">1.021</font> GiB
Labels<font color="#990000">:</font> <font color="#009900">provider</font><font color="#990000">=</font>virtualbox<font color="#990000">,</font> <font color="#009900">public</font><font color="#990000">=</font>yes<font color="#990000">,</font> <font color="#009900">storagedriver</font><font color="#990000">=</font>aufs
swarm-master<font color="#990000">:</font> <font color="#993399">192.168</font><font color="#990000">.</font><font color="#993399">99.101</font><font color="#990000">:</font><font color="#993399">2376</font>
Containers<font color="#990000">:</font> <font color="#993399">2</font>
Reserved CPUs<font color="#990000">:</font> <font color="#993399">0</font> <font color="#990000">/</font> <font color="#993399">1</font>
Reserved Memory<font color="#990000">:</font> <font color="#993399">0</font> B <font color="#990000">/</font> <font color="#993399">1.021</font> GiB
Labels<font color="#990000">:</font> <font color="#009900">provider</font><font color="#990000">=</font>virtualbox<font color="#990000">,</font> <font color="#009900">public</font><font color="#990000">=</font>no<font color="#990000">,</font> <font color="#009900">storagedriver</font><font color="#990000">=</font>aufs
CPUs<font color="#990000">:</font> <font color="#993399">4</font>
Total Memory<font color="#990000">:</font> <font color="#993399">4.086</font> GiB
Name<font color="#990000">:</font> fa2d554280ff
</tt></pre></blockquote>
<h2>Starting the Containers</h2>
<p>Now it is time to start the containers. The plan is to bring up two
database containers, Postgres and Redis, two counter web-services, and one
proxy to front the whole cluster, like this.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwcNVBDA5pVigNuFNoC6kAUBXVj8ouLPT4z0t1LIRkO0ZZObINP5U-5J-Q6OEp6ZVwJ8euWg5Kdej0BdoGYxPAdKjpiTM8WRABmJVKW_PdA-BdkpCgyaVLxf4isKKcd5cJUB2M/s1600/the-cluster.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwcNVBDA5pVigNuFNoC6kAUBXVj8ouLPT4z0t1LIRkO0ZZObINP5U-5J-Q6OEp6ZVwJ8euWg5Kdej0BdoGYxPAdKjpiTM8WRABmJVKW_PdA-BdkpCgyaVLxf4isKKcd5cJUB2M/s1600/the-cluster.png" /></a></div>
<p>Alright, let's start some containers!</p>
<h3>Databases</h3>
<p>According to the picture above I want to put the
<a href="https://hub.docker.com/_/redis/">Redis</a> container on the machine named
<code>backend1</code>, but I don't want to address it by name, instead I'm going to target
it by its labels.</p>
<p>I also want to start a <a href="https://hub.docker.com/_/postgres/">Postgres</a> container
on a machine with a <code>constraint:model==large-disk</code>.</p>
<h3>Starting Redis</h3>
<blockquote><pre><tt><i><font color="#9A1900"># Start Redis on a non-public machine with high-memory.</font></i>
$ docker run -d --name redis <font color="#990000">\</font>
--env constraint<font color="#990000">:</font>public<font color="#990000">!=</font>yes <font color="#990000">\</font>
--env constraint<font color="#990000">:</font><font color="#009900">model</font><font color="#990000">==</font>high-memory <font color="#990000">\</font>
redis
</tt></pre></blockquote>
<p>In this case, <code>constraint:public!=yes</code> is not needed but I like to add it to
avoid mistakes.</p>
<h4>Starting Postgres</h4>
<blockquote><pre><tt><i><font color="#9A1900"># Start Postgres on a non-public machine with large-disk</font></i>
$ docker run -d --name postgres <font color="#990000">\</font>
--env constraint<font color="#990000">:</font>public<font color="#990000">!=</font>yes <font color="#990000">\</font>
--env constraint<font color="#990000">:</font><font color="#009900">model</font><font color="#990000">==</font>large-disk <font color="#990000">\</font>
postgres
</tt></pre></blockquote>
<p>If this was not a VirtualBox machine I would also mount a volume,
<code>-v /var/pgdata:/var/lib/postgresql/data</code>, for the database, but this does not work
with VirtualBox.</p>
<p>OK, let's see what we have.</p>
<blockquote><pre><tt><i><font color="#9A1900"># List running containers, output slightly trimmed</font></i>
$ docker ps
CONTAINER ID IMAGE COMMAND PORTS NAMES
aa1679b3da5c postgres <font color="#FF0000">"/docker-entrypoint.s"</font> <font color="#993399">5432</font>/tcp backend2/postgres
ffa41d90f414 redis <font color="#FF0000">"/entrypoint.sh redis"</font> <font color="#993399">6379</font>/tcp backend1/redis
</tt></pre></blockquote>
<p>Nice, two running databases on the designated machines.</p>
<h3>Starting the Reverse Proxy</h3>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhNPMKwQ2jdKsXu4Pt0b_uLZcnJSNHY8oix9Bicb2Z31UlfoHhhDlBoeq_qxkCFvYOHt_HKBv6z9EyPu-BFMnnp_il74CvJFWqyLo1itNhHGcgpVK2vBoOXfpFYw_1Ux1bZHbRz/s1600/nginx.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhNPMKwQ2jdKsXu4Pt0b_uLZcnJSNHY8oix9Bicb2Z31UlfoHhhDlBoeq_qxkCFvYOHt_HKBv6z9EyPu-BFMnnp_il74CvJFWqyLo1itNhHGcgpVK2vBoOXfpFYw_1Ux1bZHbRz/s1600/nginx.png" /></a></div>
<p>Nginx is one of my favorite building blocks when it comes to building reliable
web services. Nginx provides an <a href="https://hub.docker.com/_/nginx/">official Docker image</a>,
but in this case, when I want to automatically configure Nginx when new containers
are started, I prefer to use an alternative image called <a href="https://github.com/jwilder/nginx-proxy">nginx-proxy</a>.</p>
<p>A container started from the <code>nginx-proxy</code> image, listens to events generated
by the docker engine. The engine generates events for <a href="http://docs.docker.com/engine/reference/api/docker_remote_api_v1.17/#monitor-dockers-events">all kinds of
events</a>
but all we care about here is when a container is started and stopped. If you
want to see what events are triggered from the CLI, run <code>docker events</code> in one
terminal and start and stop a few containers in another.</p>
<p>When <code>nginx-proxy</code> receives an event that a container has been started it checks
if the container has any ports <code>EXPOSE</code>d, if it does it also checks for a
<code>VIRTUAL_HOST</code> environment variable. If both these conditions are fulfilled
<code>nginx-proxy</code> re-configures its Nginx server and reloads the configuration.</p>
<p>When you now access the <code>VIRTUAL_HOST</code>, Nginx proxies the connection to your web
service. Cool!</p>
<p>Naturally, you will have to configure your DNS to point to your Nginx server.
The easiest way to do this is to configure all your services to point to it
with a wildcard record. Something like this:</p>
<blockquote><pre><tt><font color="#990000">*.</font>mysite<font color="#990000">.</font>com Host <font color="#990000">(</font>A<font color="#990000">)</font> Default xxx<font color="#990000">.</font>xxx<font color="#990000">.</font>xxx<font color="#990000">.</font>xxx
</tt></pre></blockquote>
<p>In this case, we are using VirtualBox and we can settle for changing the
<code>/etc/hosts</code> file with the IP-number of our <code>frontend</code>.</p>
<blockquote><pre><tt><i><font color="#9A1900"># /etc/hosts</font></i>
redis-counter<font color="#990000">.</font>docker <font color="#993399">192.168</font><font color="#990000">.</font><font color="#993399">99.102</font>
postgres-counter<font color="#990000">.</font>docker <font color="#993399">192.168</font><font color="#990000">.</font><font color="#993399">99.102</font>
</tt></pre></blockquote>
<p><strong>What is even more cool is that events works with Swarm</strong> and it is possible
to use the <code>nginx-proxy</code> to listen to services that are started on different
machines. All we have to do is configure it correctly.</p>
<h4>Starting Nginx-Proxy</h4>
<p><code>nginx-proxy</code> is started with configuration read from the docker client
environment variables. All the environments variables were automatically
configured when you configured the docker client to access the Swarm, above.</p>
<blockquote><pre><tt><i><font color="#9A1900"># Start nginx-proxy configured to listen to swarm events, published on port 80.</font></i>
$ docker run -d --name nginx <font color="#990000">\</font>
-v <font color="#009900">$DOCKER_CERT_PATH</font><font color="#990000">:</font><font color="#009900">$DOCKER_CERT_PATH</font> <font color="#990000">\</font>
-p <font color="#FF0000">"80:80"</font> <font color="#990000">\</font>
--env constraint<font color="#990000">:</font><font color="#009900">public</font><font color="#990000">==</font>yes <font color="#990000">\</font>
--env DOCKER_HOST <font color="#990000">\</font>
--env DOCKER_CERT_PATH <font color="#990000">\</font>
--env DOCKER_TLS_VERIFY <font color="#990000">\</font>
jwilder/nginx-proxy
</tt></pre></blockquote>
<p>OK, we are almost done. Now it is time to start the web services.</p>
<h3>Starting Web Services</h3>
<p>As a web service I'm going to use a <a href="https://hub.docker.com/r/andersjanmyr/counter/">simple counter image</a> since it can use both Postgres and Redis as backend.
I want to start the web services on the same server as the databases since this
allows me to use <code>--link</code> to connect to the container and it will speed up the
data access. To do this I can use an affinity constraint: <code>--env
affinity:container==*redis*</code>.</p>
<blockquote><pre><tt><i><font color="#9A1900"># Start a counter close to the container named redis and link to it.</font></i>
$ docker run -d --name redis-counter <font color="#990000">\</font>
-p <font color="#993399">80</font> <font color="#990000">\</font>
--link redis <font color="#990000">\</font>
--env affinity<font color="#990000">:</font><font color="#009900">container</font><font color="#990000">==*</font>redis<font color="#990000">*</font> <font color="#990000">\</font>
--env <font color="#009900">REDIS_URL</font><font color="#990000">=</font>redis<font color="#990000">:</font><font color="#993399">6379</font> <font color="#990000">\</font>
--env <font color="#009900">VIRTUAL_HOST</font><font color="#990000">=</font>redis-counter<font color="#990000">.</font>docker <font color="#990000">\</font>
andersjanmyr/counter
</tt></pre></blockquote>
<p>The affinity constraint is not really necessary since affinity constraints are
automatically generated by Swarm when <code>--link</code> is present as you can see when
we start the <code>postgres-counter</code>.</p>
<blockquote><pre><tt><i><font color="#9A1900"># Start a counter close to the container named postgres and link to it.</font></i>
$ docker run -d --name postgres-counter <font color="#990000">\</font>
-p <font color="#993399">80</font> <font color="#990000">\</font>
--link postgres <font color="#990000">\</font>
--env <font color="#009900">POSTGRES_URL</font><font color="#990000">=</font>postgres<font color="#990000">:</font>//postgres@postgres <font color="#990000">\</font>
--env <font color="#009900">VIRTUAL_HOST</font><font color="#990000">=</font>postgres-counter<font color="#990000">.</font>docker <font color="#990000">\</font>
andersjanmyr/counter
</tt></pre></blockquote>
<p>Browse to <a href="http://redis-counter.docker">http://redis-counter.docker</a> or
<a href="http:/postgres-counter.docker">http:/postgres-counter.docker</a> and
you should see your services up and running.</p>
<p>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi6QvMBPc_yp3AKcHiqxe9n-TitJpPXNksjwe5ywuBsDIGTBHma68NfgeY7ab1q7NneBLTj6JjfbzFHQH-b-72IeTpB9ZCplyaUYHko6gKin8JFNDGfoK6MdNtmYkf-r1CT4dvb/s1600/redis-counter.png" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi6QvMBPc_yp3AKcHiqxe9n-TitJpPXNksjwe5ywuBsDIGTBHma68NfgeY7ab1q7NneBLTj6JjfbzFHQH-b-72IeTpB9ZCplyaUYHko6gKin8JFNDGfoK6MdNtmYkf-r1CT4dvb/s200/redis-counter.png" /></a><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEikD2Jq9H4KpVIMShBkYuuBDTo47yUEc3cGA1H-PCFQc2NoluuT5jDTCzjJkv9Xqmxhz8jPMTinLfhtFcGq97104H_dkI9T2qkZaWyPxhU-TP1f3CMBFxeSkmRbWB9TN_hafFr3/s1600/postgres-counter.png" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEikD2Jq9H4KpVIMShBkYuuBDTo47yUEc3cGA1H-PCFQc2NoluuT5jDTCzjJkv9Xqmxhz8jPMTinLfhtFcGq97104H_dkI9T2qkZaWyPxhU-TP1f3CMBFxeSkmRbWB9TN_hafFr3/s200/postgres-counter.png" /></a>
</p>
<h2>Summary</h2>
<p>Here's an illustration of our current setup:</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjWRFEnG7Mtc5LpC7CJvTuU3lw4EbxdbPMGUo6SGglMkP4DH5tPlzMrWQf_hCYj4NBNvE5KG6NCe4-AqC2i4msw_PeyO3QzXY5j-xEQtu2VNeKQoFRzbdKPMZNv3gEav4ptzsBJ/s1600/swarm-complete.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjWRFEnG7Mtc5LpC7CJvTuU3lw4EbxdbPMGUo6SGglMkP4DH5tPlzMrWQf_hCYj4NBNvE5KG6NCe4-AqC2i4msw_PeyO3QzXY5j-xEQtu2VNeKQoFRzbdKPMZNv3gEav4ptzsBJ/s1600/swarm-complete.png" /></a></div>
<p>And here is a listing of all the containers on their respective machines.</p>
<blockquote><pre><tt>$ docker ps
CONTAINER ID IMAGE COMMAND PORTS NAMES
b3869a89e76c andersjanmyr/counter <font color="#FF0000">"/counter-linux"</font> <font color="#993399">192.168</font><font color="#990000">.</font><font color="#993399">99.104</font><font color="#990000">:</font><font color="#993399">32768</font>-<font color="#990000">></font><font color="#993399">80</font>/tcp backend2/postgres-counter
cff69b6f970d andersjanmyr/counter <font color="#FF0000">"/counter-linux"</font> <font color="#993399">192.168</font><font color="#990000">.</font><font color="#993399">99.103</font><font color="#990000">:</font><font color="#993399">32768</font>-<font color="#990000">></font><font color="#993399">80</font>/tcp backend1/redis-counter
64af31135c26 jwilder/nginx-proxy <font color="#FF0000">"/app/docker-entrypoi"</font> <font color="#993399">443</font>/tcp<font color="#990000">,</font> <font color="#993399">192.168</font><font color="#990000">.</font><font color="#993399">99.102</font><font color="#990000">:</font><font color="#993399">80</font>-<font color="#990000">></font><font color="#993399">80</font>/tcp frontend/nginx
aa1679b3da5c postgres <font color="#FF0000">"/docker-entrypoint.s"</font> <font color="#993399">5432</font>/tcp backend2/postgres<font color="#990000">,</font>backend2/postgres-counter/postgres
ffa41d90f414 redis <font color="#FF0000">"/entrypoint.sh redis"</font> <font color="#993399">6379</font>/tcp backend1/redis<font color="#990000">,</font>backend1/redis-counter/redis
</tt></pre></blockquote>
<p>May the Swarm be with you! :D</p>
Anders Janmyrhttp://www.blogger.com/profile/16081537018884349622noreply@blogger.com4tag:blogger.com,1999:blog-34049130.post-77370777851596751612015-07-02T20:11:00.000+02:002015-07-03T14:26:58.184+02:00Extending CloudFormation with Lambda-Backed Custom Resources<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiuVe8K2t7QOLFJzNdIwdAoevE4BbAehTcs7QRsdqyLB5mg1-eNjyRAsJonNWmXV_tApfsEscsm1U3hbH9UdhNR4Svq0VvAajQE41wMaZ69DguoOrfY6XPirmlRL9NhhiDI-4QQ/s1600/aws-cloud.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiuVe8K2t7QOLFJzNdIwdAoevE4BbAehTcs7QRsdqyLB5mg1-eNjyRAsJonNWmXV_tApfsEscsm1U3hbH9UdhNR4Svq0VvAajQE41wMaZ69DguoOrfY6XPirmlRL9NhhiDI-4QQ/s200/aws-cloud.png" /></a></div>
<p>CloudFormation stacks are very nice for setting up complete server environments.
But, there are a few problems.</p>
<ul>
<li>Keeping all resources in one stack may not be possible since some resources,
such as Kineses, may be needed by multiple stacks.</li>
<li>If your stack depends on external resources you have to maintain these
resources separately and send them as parameters when creating new stacks.</li>
<li>If you have multiple accounts or environments you have to maintain one
configuration for each.</li>
</ul>
<p>Using Custom Resources to extend the native functionality of CloudFormation solves
these problems.</p>
<p>At Sony Mobile in Lund we make heavy use of Lambda-backed Custom Resources. We
use them for depending on other stacks, getting info about VPC, Route53,
certificates and AMIs. We also have a resource for getting Elasticache
endpoints to our Redis services since CloudFormation does not provide it.</p>
<p>The code in this article is based on code from <a href="http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-custom-resources-lambda.html">Amazon Lambda-backed Custom
Resources</a></p>
<h2>The Custom Resource Invocation</h2>
<p>When CloudFormation invokes a Custom Resource it send a request which looks like
this:</p>
<blockquote><pre><tt><font color="#FF0000">{</font>
<font color="#FF0000">"RequestType"</font><font color="#990000">:</font> <font color="#FF0000">"Create"</font><font color="#990000">,</font>
<font color="#FF0000">"ServiceToken"</font><font color="#990000">:</font> <font color="#FF0000">"arn:aws:lambda:...:function:route53Dependency"</font><font color="#990000">,</font>
<font color="#FF0000">"ResponseURL"</font><font color="#990000">:</font> <font color="#FF0000">"https://cloudformation-custom-resource..."</font><font color="#990000">,</font>
<font color="#FF0000">"StackId"</font><font color="#990000">:</font> <font color="#FF0000">"arn:aws:cloudformation:eu-west-1:..."</font><font color="#990000">,</font>
<font color="#FF0000">"RequestId"</font><font color="#990000">:</font> <font color="#FF0000">"afd8d7c5-9376-4013-8b3b-307517b8719e"</font><font color="#990000">,</font>
<font color="#FF0000">"LogicalResourceId"</font><font color="#990000">:</font> <font color="#FF0000">"Route53"</font><font color="#990000">,</font>
<font color="#FF0000">"ResourceType"</font><font color="#990000">:</font> <font color="#FF0000">"Custom::Route53Dependency"</font><font color="#990000">,</font>
<font color="#FF0000">"ResourceProperties"</font><font color="#990000">:</font> <font color="#FF0000">{</font>
<font color="#FF0000">"ServiceToken"</font><font color="#990000">:</font> <font color="#FF0000">"arn:aws:lambda:...:function:route53Dependency"</font><font color="#990000">,</font>
<font color="#FF0000">"DomainName"</font><font color="#990000">:</font> <font color="#FF0000">"example.com"</font>
<font color="#FF0000">}</font>
<font color="#FF0000">}</font>
</tt></pre></blockquote>
<p>The dots, (...), are random digits and account numbers and will be different
per request and account. The interesting parts of this request are:</p>
<ul>
<li><code>RequestType</code> - can have the values <code>Create</code>, <code>Update</code> and <code>Delete</code>.</li>
<li><code>ResponseURL</code> - the URL to <code>PUT</code> the response to.</li>
<li><code>ResourceProperties</code> - the properties sent by the configuration in the
CloudFormation resource declaration. You can ignore the <code>ServiceToken</code>, it is
used internally by CloudFormation to find your function. <code>DomainName</code> is
interesting since it contains the name of the domain we want to lookup.</li>
</ul>
<h2>The Custom Resource Declaration</h2>
<p>The Custom Resource Declaration corresponding to the above invocation looks
like this:</p>
<blockquote><pre><tt><font color="#FF0000">"Resources"</font><font color="#990000">:</font> <font color="#FF0000">{</font>
<font color="#FF0000">"Route53"</font><font color="#990000">:</font> <font color="#FF0000">{</font>
<font color="#FF0000">"Type"</font><font color="#990000">:</font> <font color="#FF0000">"Custom::Route53Dependency"</font><font color="#990000">,</font>
<font color="#FF0000">"Properties"</font><font color="#990000">:</font> <font color="#FF0000">{</font>
<font color="#FF0000">"ServiceToken"</font><font color="#990000">:</font> <font color="#FF0000">{</font> <font color="#FF0000">"Fn::Join"</font><font color="#990000">:</font> <font color="#990000">[</font> <font color="#FF0000">""</font><font color="#990000">,</font> <font color="#990000">[</font>
<font color="#FF0000">"arn:aws:lambda:"</font><font color="#990000">,</font>
<font color="#FF0000">{</font> <font color="#FF0000">"Ref"</font><font color="#990000">:</font> <font color="#FF0000">"AWS::Region"</font> <font color="#FF0000">}</font><font color="#990000">,</font>
<font color="#FF0000">":"</font><font color="#990000">,</font>
<font color="#FF0000">{</font> <font color="#FF0000">"Ref"</font><font color="#990000">:</font> <font color="#FF0000">"AWS::AccountId"</font> <font color="#FF0000">}</font><font color="#990000">,</font>
<font color="#FF0000">":function:route53Dependency"</font>
<font color="#990000">]</font> <font color="#990000">]</font> <font color="#FF0000">}</font><font color="#990000">,</font>
<font color="#FF0000">"DomainName"</font><font color="#990000">:</font> <font color="#FF0000">{</font> <font color="#FF0000">"Ref"</font><font color="#990000">:</font> <font color="#FF0000">"DomainName"</font> <font color="#FF0000">}</font>
<font color="#FF0000">}</font>
<font color="#FF0000">}</font>
<font color="#FF0000">}</font>
</tt></pre></blockquote>
<p>The interesting parts here are:</p>
<ul>
<li><code>Route53</code> - the name of the resource inside the template.</li>
<li><code>Type</code> - The <code>Custom::</code> part of the type identifies this as a Custom
Resource, the rest, Route53Dependency, is documentation.</li>
<li><code>ServiceToken</code> - identifies the function with the current region, account and
the name <code>route53Dependency</code>. This function name corresponds to the name of
the function used when creating the function with <code>aws lambda
create-function --function-name route53Dependency</code></li>
<li><code>DomainName</code> - is a custom parameter which we will read from the event and use
to lookup the domain in Route53.</li>
</ul>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgs85AAORc62SX5jfp22-QS5Y9WdyLLzhNUjXLGEeWKqwQgHfzSzVgp-n8lvXpwXVMG1JSUiDpRBvD6NKYA-nX6OSTig5Z-EW3tGf27vJaspq_prsXZqg7aa4-QtAF8qtPTtK9i/s1600/Lambda.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgs85AAORc62SX5jfp22-QS5Y9WdyLLzhNUjXLGEeWKqwQgHfzSzVgp-n8lvXpwXVMG1JSUiDpRBvD6NKYA-nX6OSTig5Z-EW3tGf27vJaspq_prsXZqg7aa4-QtAF8qtPTtK9i/s200/Lambda.png" /></a></div>
<h2>The Lambda Custom Resource Function</h2>
<p>A Custom Resource Lambda can be split into three major parts.</p>
<ul>
<li>The handler handles the event and invokes the domain specific function.</li>
<li>The domain specific function calls the AWS function and gets the requested
information.</li>
<li>The response is sent back to CloudFormation by <code>PUT</code>ing to the
provided <code>ResponseURL</code></li>
</ul>
<p>The handler and response code is almost the same for all our Custom Resources.</p>
<h3>The handler</h3>
<blockquote><pre><tt><i><font color="#9A1900">// The handler</font></i>
route53Dependency<font color="#990000">.</font>handler <font color="#990000">=</font> <b><font color="#0000FF">function</font></b><font color="#990000">(</font>event<font color="#990000">,</font> context<font color="#990000">)</font> <font color="#FF0000">{</font>
console<font color="#990000">.</font><b><font color="#000000">log</font></b><font color="#990000">(</font>JSON<font color="#990000">.</font><b><font color="#000000">stringify</font></b><font color="#990000">(</font>event<font color="#990000">,</font> <b><font color="#0000FF">null</font></b><font color="#990000">,</font> <font color="#FF0000">' '</font><font color="#990000">));</font>
<b><font color="#0000FF">if</font></b> <font color="#990000">(</font>event<font color="#990000">.</font>RequestType <font color="#990000">==</font> <font color="#FF0000">'Delete'</font><font color="#990000">)</font> <font color="#FF0000">{</font>
<b><font color="#0000FF">return</font></b> <b><font color="#000000">sendResponse</font></b><font color="#990000">(</font>event<font color="#990000">,</font> context<font color="#990000">,</font> <font color="#FF0000">"SUCCESS"</font><font color="#990000">);</font>
<font color="#FF0000">}</font>
<b><font color="#000000">route53Dependency</font></b><font color="#990000">(</font>event<font color="#990000">.</font>ResourceProperties<font color="#990000">,</font> <b><font color="#0000FF">function</font></b><font color="#990000">(</font>err<font color="#990000">,</font> result<font color="#990000">)</font> <font color="#FF0000">{</font>
<b><font color="#0000FF">var</font></b> status <font color="#990000">=</font> err <font color="#990000">?</font> <font color="#FF0000">'FAILED'</font> <font color="#990000">:</font> <font color="#FF0000">'SUCCESS'</font><font color="#990000">;</font>
<b><font color="#0000FF">return</font></b> <b><font color="#000000">sendResponse</font></b><font color="#990000">(</font>event<font color="#990000">,</font> context<font color="#990000">,</font> status<font color="#990000">,</font> result<font color="#990000">,</font> err<font color="#990000">);</font>
<font color="#FF0000">}</font><font color="#990000">);</font>
<font color="#FF0000">}</font><font color="#990000">;</font>
</tt></pre></blockquote>
<p>The handler starts by logging the event, nice to have for debugging. It ignores
<code>Delete</code>-events since our resources are not creating anything, only looking
things up. It then invokes the domain function with <code>event.ResourceProperties</code>,
checks the result and uses <code>sendResponse</code> to send the reply.</p>
<p>Lambda functions have to be invoked as properties, in this case <code>handler</code>.
I usually set the handler as a property on the domain function. This is just my
preference, it is not required.</p>
<h3>The Domain Function</h3>
<blockquote><pre><tt><i><font color="#9A1900">// The domain function</font></i>
<b><font color="#0000FF">function</font></b> <b><font color="#000000">route53Dependency</font></b><font color="#990000">(</font>properties<font color="#990000">,</font> callback<font color="#990000">)</font> <font color="#FF0000">{</font>
<b><font color="#0000FF">if</font></b> <font color="#990000">(!</font>properties<font color="#990000">.</font>DomainName<font color="#990000">)</font>
<b><font color="#000000">callback</font></b><font color="#990000">(</font><font color="#FF0000">"Domain name not specified"</font><font color="#990000">);</font>
<b><font color="#0000FF">var</font></b> aws <font color="#990000">=</font> <b><font color="#000000">require</font></b><font color="#990000">(</font><font color="#FF0000">"aws-sdk"</font><font color="#990000">);</font>
<b><font color="#0000FF">var</font></b> route53 <font color="#990000">=</font> <b><font color="#0000FF">new</font></b> aws<font color="#990000">.</font><b><font color="#000000">Route53</font></b><font color="#990000">();</font>
console<font color="#990000">.</font><b><font color="#000000">log</font></b><font color="#990000">(</font><font color="#FF0000">'route53Dependency'</font><font color="#990000">,</font> properties<font color="#990000">);</font>
route53<font color="#990000">.</font><b><font color="#000000">listHostedZones</font></b><font color="#990000">(</font><font color="#FF0000">{}</font><font color="#990000">,</font> <b><font color="#0000FF">function</font></b><font color="#990000">(</font>err<font color="#990000">,</font> data<font color="#990000">)</font> <font color="#FF0000">{</font>
console<font color="#990000">.</font><b><font color="#000000">log</font></b><font color="#990000">(</font><font color="#FF0000">'listHostedZones'</font><font color="#990000">,</font> err<font color="#990000">,</font> data<font color="#990000">);</font>
<b><font color="#0000FF">if</font></b> <font color="#990000">(</font>err<font color="#990000">)</font>
<b><font color="#0000FF">return</font></b> <b><font color="#000000">callback</font></b><font color="#990000">(</font>err<font color="#990000">);</font>
<b><font color="#0000FF">var</font></b> zones <font color="#990000">=</font> data<font color="#990000">.</font>HostedZones<font color="#990000">;</font>
<b><font color="#0000FF">var</font></b> matching <font color="#990000">=</font> zones<font color="#990000">.</font><b><font color="#000000">filter</font></b><font color="#990000">(</font><b><font color="#0000FF">function</font></b><font color="#990000">(</font>zone<font color="#990000">)</font> <font color="#FF0000">{</font>
<b><font color="#0000FF">return</font></b> zone<font color="#990000">.</font>Name <font color="#990000">==</font> properties<font color="#990000">.</font>DomainName <font color="#990000">+</font> <font color="#FF0000">'.'</font><font color="#990000">;</font>
<font color="#FF0000">}</font><font color="#990000">);</font>
<b><font color="#0000FF">if</font></b> <font color="#990000">(</font>matching<font color="#990000">.</font>length <font color="#990000">!=</font> <font color="#993399">1</font><font color="#990000">)</font>
<b><font color="#0000FF">return</font></b> <b><font color="#000000">callback</font></b><font color="#990000">(</font><font color="#FF0000">'Exactly one matching zone is allowed '</font> <font color="#990000">+</font> zones<font color="#990000">);</font>
<b><font color="#0000FF">var</font></b> match <font color="#990000">=</font> matching<font color="#990000">[</font><font color="#993399">0</font><font color="#990000">];</font>
<b><font color="#0000FF">delete</font></b> match<font color="#990000">.</font>Config<font color="#990000">;</font>
<b><font color="#0000FF">delete</font></b> match<font color="#990000">.</font>CallerReference<font color="#990000">;</font>
match<font color="#990000">.</font>Id <font color="#990000">=</font> match<font color="#990000">.</font>Id<font color="#990000">.</font><b><font color="#000000">split</font></b><font color="#990000">(</font><font color="#FF0000">'/'</font><font color="#990000">)[</font><font color="#993399">2</font><font color="#990000">];</font>
match<font color="#990000">.</font>Name <font color="#990000">=</font> match<font color="#990000">.</font>Name<font color="#990000">.</font><b><font color="#000000">substring</font></b><font color="#990000">(</font><font color="#993399">0</font><font color="#990000">,</font> match<font color="#990000">.</font>Name<font color="#990000">.</font>length<font color="#990000">-</font><font color="#993399">1</font><font color="#990000">);</font>
<b><font color="#0000FF">return</font></b> <b><font color="#000000">callback</font></b><font color="#990000">(</font><b><font color="#0000FF">null</font></b><font color="#990000">,</font> match<font color="#990000">);</font>
<font color="#FF0000">}</font><font color="#990000">);</font>
<font color="#FF0000">}</font>
</tt></pre></blockquote>
<p>The domain function starts out with checking for required properties and calls
the callback with an error in case they are not valid or available.</p>
<p>We require the SDK and invoke <code>listHostedZones</code>, check that exactly one domain
matches and return the object after cleaning it up. Custom Resources only
supports returning string values and if you try to return objects such as
<code>Config</code> above. In this case we don't care about this value and just delete it.</p>
<h3>The Response</h3>
<p>To send the response back to CloudFormation we PUT some JSON to the
<code>ResponseURL</code> provided in the request. The response looks like this:</p>
<blockquote><pre><tt><font color="#FF0000">{</font>
<font color="#FF0000">"StackId"</font><font color="#990000">:</font> <font color="#FF0000">"arn:aws:cloudformation:eu-west-1:..."</font><font color="#990000">,</font>
<font color="#FF0000">"RequestId"</font><font color="#990000">:</font> <font color="#FF0000">'e4d1ab88-1b2c-402f-b083-1966f5806064'</font><font color="#990000">,</font>
<font color="#FF0000">"LogicalResourceId"</font><font color="#990000">:</font> <font color="#FF0000">'Route53'</font><font color="#990000">,</font>
<font color="#FF0000">"PhysicalResourceId"</font><font color="#990000">:</font> <font color="#FF0000">'2015/05/28/00395b017f72444791fb12b988f4aeab'</font><font color="#990000">,</font>
<font color="#FF0000">"Status"</font><font color="#990000">:</font> <font color="#FF0000">'SUCCESS'</font><font color="#990000">,</font>
<font color="#FF0000">"Reason"</font><font color="#990000">:</font> <font color="#FF0000">' Details in CloudWatch Log: 2015/05/28/...'</font><font color="#990000">,</font>
<font color="#FF0000">"Data"</font><font color="#990000">:</font> <font color="#FF0000">{</font>
<font color="#FF0000">"Id"</font><font color="#990000">:</font> <font color="#FF0000">'ZEHTTI1S7FAPK'</font><font color="#990000">,</font>
<font color="#FF0000">"Name"</font><font color="#990000">:</font> <font color="#FF0000">'backend.lifelog-dev.sonymobile.com'</font><font color="#990000">,</font>
<font color="#FF0000">"ResourceRecordSetCount"</font><font color="#990000">:</font> <font color="#FF0000">"22"</font>
<font color="#FF0000">}</font>
<font color="#FF0000">}</font>
</tt></pre></blockquote>
<p>The relevant parts of the response are the last three:</p>
<ul>
<li>Status - signals <code>SUCCESS</code> of <code>FAILED</code>.</li>
<li>Reason - information about the request, such as where to find the log or the
error if one occurred.</li>
<li>Data - is the data provided by our domain function.</li>
</ul>
<p>We don't have to care about the rest of the parameters since we can just echo
back the parameters which were passed to us in the request.</p>
<blockquote><pre><tt><i><font color="#9A1900">// sendResponse</font></i>
<b><font color="#0000FF">function</font></b> <b><font color="#000000">sendResponse</font></b><font color="#990000">(</font>event<font color="#990000">,</font> context<font color="#990000">,</font> status<font color="#990000">,</font> data<font color="#990000">,</font> err<font color="#990000">)</font> <font color="#FF0000">{</font>
<b><font color="#0000FF">var</font></b> reason <font color="#990000">=</font> err <font color="#990000">?</font> err<font color="#990000">.</font>message <font color="#990000">:</font> <font color="#FF0000">''</font><font color="#990000">;</font>
<b><font color="#0000FF">var</font></b> responseBody <font color="#990000">=</font> <font color="#FF0000">{</font>
StackId<font color="#990000">:</font> event<font color="#990000">.</font>StackId<font color="#990000">,</font>
RequestId<font color="#990000">:</font> event<font color="#990000">.</font>RequestId<font color="#990000">,</font>
LogicalResourceId<font color="#990000">:</font> event<font color="#990000">.</font>LogicalResourceId<font color="#990000">,</font>
PhysicalResourceId<font color="#990000">:</font> context<font color="#990000">.</font>logStreamName<font color="#990000">,</font>
Status<font color="#990000">:</font> status<font color="#990000">,</font>
Reason<font color="#990000">:</font> reason <font color="#990000">+</font> <font color="#FF0000">" See details in CloudWatch Log: "</font> <font color="#990000">+</font> context<font color="#990000">.</font>logStreamName<font color="#990000">,</font>
Data<font color="#990000">:</font> data
<font color="#FF0000">}</font><font color="#990000">;</font>
console<font color="#990000">.</font><b><font color="#000000">log</font></b><font color="#990000">(</font><font color="#FF0000">"RESPONSE:</font><font color="#CC33CC">\n</font><font color="#FF0000">"</font><font color="#990000">,</font> responseBody<font color="#990000">);</font>
<b><font color="#0000FF">var</font></b> json <font color="#990000">=</font> JSON<font color="#990000">.</font><b><font color="#000000">stringify</font></b><font color="#990000">(</font>responseBody<font color="#990000">);</font>
<b><font color="#0000FF">var</font></b> https <font color="#990000">=</font> <b><font color="#000000">require</font></b><font color="#990000">(</font><font color="#FF0000">"https"</font><font color="#990000">);</font>
<b><font color="#0000FF">var</font></b> url <font color="#990000">=</font> <b><font color="#000000">require</font></b><font color="#990000">(</font><font color="#FF0000">"url"</font><font color="#990000">);</font>
<b><font color="#0000FF">var</font></b> parsedUrl <font color="#990000">=</font> url<font color="#990000">.</font><b><font color="#000000">parse</font></b><font color="#990000">(</font>event<font color="#990000">.</font>ResponseURL<font color="#990000">);</font>
<b><font color="#0000FF">var</font></b> options <font color="#990000">=</font> <font color="#FF0000">{</font>
hostname<font color="#990000">:</font> parsedUrl<font color="#990000">.</font>hostname<font color="#990000">,</font>
port<font color="#990000">:</font> <font color="#993399">443</font><font color="#990000">,</font>
path<font color="#990000">:</font> parsedUrl<font color="#990000">.</font>path<font color="#990000">,</font>
method<font color="#990000">:</font> <font color="#FF0000">"PUT"</font><font color="#990000">,</font>
headers<font color="#990000">:</font> <font color="#FF0000">{</font>
<font color="#FF0000">"content-type"</font><font color="#990000">:</font> <font color="#FF0000">""</font><font color="#990000">,</font>
<font color="#FF0000">"content-length"</font><font color="#990000">:</font> json<font color="#990000">.</font>length
<font color="#FF0000">}</font>
<font color="#FF0000">}</font><font color="#990000">;</font>
<b><font color="#0000FF">var</font></b> request <font color="#990000">=</font> https<font color="#990000">.</font><b><font color="#000000">request</font></b><font color="#990000">(</font>options<font color="#990000">,</font> <b><font color="#0000FF">function</font></b><font color="#990000">(</font>response<font color="#990000">)</font> <font color="#FF0000">{</font>
console<font color="#990000">.</font><b><font color="#000000">log</font></b><font color="#990000">(</font><font color="#FF0000">"STATUS: "</font> <font color="#990000">+</font> response<font color="#990000">.</font>statusCode<font color="#990000">);</font>
console<font color="#990000">.</font><b><font color="#000000">log</font></b><font color="#990000">(</font><font color="#FF0000">"HEADERS: "</font> <font color="#990000">+</font> JSON<font color="#990000">.</font><b><font color="#000000">stringify</font></b><font color="#990000">(</font>response<font color="#990000">.</font>headers<font color="#990000">));</font>
context<font color="#990000">.</font><b><font color="#000000">done</font></b><font color="#990000">(</font><b><font color="#0000FF">null</font></b><font color="#990000">,</font> data<font color="#990000">);</font>
<font color="#FF0000">}</font><font color="#990000">);</font>
request<font color="#990000">.</font><b><font color="#000000">on</font></b><font color="#990000">(</font><font color="#FF0000">"error"</font><font color="#990000">,</font> <b><font color="#0000FF">function</font></b><font color="#990000">(</font>error<font color="#990000">)</font> <font color="#FF0000">{</font>
console<font color="#990000">.</font><b><font color="#000000">log</font></b><font color="#990000">(</font><font color="#FF0000">"sendResponse Error:</font><font color="#CC33CC">\n</font><font color="#FF0000">"</font><font color="#990000">,</font> error<font color="#990000">);</font>
context<font color="#990000">.</font><b><font color="#000000">done</font></b><font color="#990000">(</font>error<font color="#990000">);</font>
<font color="#FF0000">}</font><font color="#990000">);</font>
request<font color="#990000">.</font><b><font color="#000000">on</font></b><font color="#990000">(</font><font color="#FF0000">"end"</font><font color="#990000">,</font> <b><font color="#0000FF">function</font></b><font color="#990000">()</font> <font color="#FF0000">{</font>
console<font color="#990000">.</font><b><font color="#000000">log</font></b><font color="#990000">(</font><font color="#FF0000">"end"</font><font color="#990000">);</font>
<font color="#FF0000">}</font><font color="#990000">);</font>
request<font color="#990000">.</font><b><font color="#000000">write</font></b><font color="#990000">(</font>json<font color="#990000">);</font>
request<font color="#990000">.</font><b><font color="#000000">end</font></b><font color="#990000">();</font>
<font color="#FF0000">}</font>
</tt></pre></blockquote>
<p>We start out by creating the response object by using the values from the
request and our values for <code>status</code>, <code>data</code> and <code>err</code>.</p>
<p>We parse the <code>ResponseURL</code> and send the request with <code>https.request</code> back to
CloudFormation.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQKufoMR76Hp4FWbzFe9ShVfYdNgbiG_uMNpLrEOgg6lBi02O_7-yCuWB2g5Q0IykjRIQgBw58Iiu-Y0xkiLEL_EEh636faNWc386dYE2w0xirp1Tal3g5ouUgzI396nvr313G/s1600/cloudformation.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQKufoMR76Hp4FWbzFe9ShVfYdNgbiG_uMNpLrEOgg6lBi02O_7-yCuWB2g5Q0IykjRIQgBw58Iiu-Y0xkiLEL_EEh636faNWc386dYE2w0xirp1Tal3g5ouUgzI396nvr313G/s200/cloudformation.jpg" /></a></div>
<h3>Using the Properties in CloudFormation</h3>
<p>To use the properties in a CloudFormation template, we use the built-in
<code>Fn::GetAtt</code> function.</p>
<blockquote><pre><tt><i><font color="#9A1900">// Example Usage</font></i>
<font color="#FF0000">"Outputs"</font><font color="#990000">:</font> <font color="#FF0000">{</font>
<font color="#FF0000">"Route53Id"</font><font color="#990000">:</font> <font color="#FF0000">{</font>
<font color="#FF0000">"Value"</font><font color="#990000">:</font> <font color="#FF0000">{</font>
<font color="#FF0000">"Fn::GetAtt"</font><font color="#990000">:</font> <font color="#990000">[</font> <font color="#FF0000">"Route53"</font><font color="#990000">,</font> <font color="#FF0000">"Id"</font> <font color="#990000">]</font>
<font color="#FF0000">}</font><font color="#990000">,</font>
<font color="#FF0000">"Description"</font><font color="#990000">:</font> <font color="#FF0000">"Route53 Id"</font>
<font color="#FF0000">}</font><font color="#990000">,</font>
<font color="#FF0000">"Route53Name"</font><font color="#990000">:</font> <font color="#FF0000">{</font>
<font color="#FF0000">"Value"</font><font color="#990000">:</font> <font color="#FF0000">{</font>
<font color="#FF0000">"Fn::GetAtt"</font><font color="#990000">:</font> <font color="#990000">[</font> <font color="#FF0000">"Route53"</font><font color="#990000">,</font> <font color="#FF0000">"Name"</font> <font color="#990000">]</font>
<font color="#FF0000">}</font><font color="#990000">,</font>
<font color="#FF0000">"Description"</font><font color="#990000">:</font> <font color="#FF0000">"Route53 Name"</font>
<font color="#FF0000">}</font><font color="#990000">,</font>
<font color="#FF0000">"Route53Count"</font><font color="#990000">:</font> <font color="#FF0000">{</font>
<font color="#FF0000">"Value"</font><font color="#990000">:</font> <font color="#FF0000">{</font>
<font color="#FF0000">"Fn::GetAtt"</font><font color="#990000">:</font> <font color="#990000">[</font> <font color="#FF0000">"Route53"</font><font color="#990000">,</font> <font color="#FF0000">"ResourceRecordSetCount"</font> <font color="#990000">]</font>
<font color="#FF0000">}</font><font color="#990000">,</font>
<font color="#FF0000">"Description"</font><font color="#990000">:</font> <font color="#FF0000">"Route53 Count"</font>
<font color="#FF0000">}</font>
<font color="#FF0000">}</font>
</tt></pre></blockquote>
<h2>Available Custom Resources</h2>
<p>We have implemented the following Custom Resources.</p>
<h3>Elasticache Dependency</h3>
<p>When CloudFormation creates a Redis-backed Elasticache Cluster it does not
provide the endpoints to the stack. This forces us to write logic in the client
to look up the endpoints or to look them up manually and provide them as
configuration. <code>elasticacheDependency</code> gets information about Elasticache
clusters including endpoints.</p>
<h3>Image Dependency</h3>
<p><code>imageDependency</code> looks up information about an AMI by name. It is much easier to
read an image name instead of an AMI ID.</p>
<h3>Route53 Dependency</h3>
<p><code>route53Dependency</code> looks up information about hosted zone by domain name.
Again, nicer to have than a cryptic zone id.</p>
<h3>VPC Dependency</h3>
<p><code>vpcDependency</code> looks up information about a VPC by name including ID and subnet
information.</p>
<h3>Certificate Dependency</h3>
<p><code>certificateDependency</code> looks up a certificate by name.</p>
<h3>Stack Dependency</h3>
<p><code>stackDependency</code> looks up the outputs from another stack by name. It provides
the outputs as variables to the resources and also includes an extra property
called <code>Environment</code>.</p>
<p>The <code>Environment</code> property contains all the outputs from the stack formatted as
a Unix <code>env-file</code>, <code>(Property1=Value\nProperty2=Value\n)</code>. This can be used to
provide the parameters to the instance by saving them to an environment file
and, if you use Docker, to provide them to the container with <code>docker run
--env-file</code></p>
<h2>Open Source</h2>
<p>All our custom resources are open sourced and can be accessed at the <a href="https://github.com/sonyxperiadev/amazon-custom-resources">Sony
Experia Dev Github account</a>.</p>
<p>The following script are available for each resource:</p>
<ul>
<li><code>create-role.sh</code> - creates the necessary roles for using the Lambda.</li>
<li><code>deploy-lambda.sh</code> - creates or updates the Lambda as needed.</li>
<li><code>invoke-lambda.sh</code> - invokes the lambda from the command line.</li>
<li><code>invoke-test-stack.sh</code> - creates or deletes a test-stack showing how to use
the function with CloudFormation.</li>
</ul>
<h2>Conclusion</h2>
<p>CloudFormation extended with Custom Resources is a powerful tool that enables
us to setup almost all our AWS resources. Currently we are not deploying our
Lambdas via CloudFormation but we are working on it.
<a href="https://github.com/sonyxperiadev/amazon-custom-resources">Stay tuned!</a></p>
Anders Janmyrhttp://www.blogger.com/profile/16081537018884349622noreply@blogger.com0tag:blogger.com,1999:blog-34049130.post-37843323479918944952015-03-19T10:00:00.000+01:002015-03-21T09:57:49.861+01:00A Not Very Short Introduction to Docker
<p>This is the notes that accompany my presentation called <a href="http://www.slideshare.net/andersjanmyr/docker-the-future-of-devops">Docker, the Future of
DevOps</a>.
It turned out, quite fittingly, to be a whale-sized article :).</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzdhT3FwjYj1MJycKlZklaw9gAeMNl5TbAc31RUf6-HUM9EQWDpPEnIgy1nHLeSth53gESaXHyaABi_zuCCLFyNHmC_OGdW8XbPcbyIlC09HLbICw2H8Ayu9kEf9kOWU-dcq6-/s1600/docker.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzdhT3FwjYj1MJycKlZklaw9gAeMNl5TbAc31RUf6-HUM9EQWDpPEnIgy1nHLeSth53gESaXHyaABi_zuCCLFyNHmC_OGdW8XbPcbyIlC09HLbICw2H8Ayu9kEf9kOWU-dcq6-/s320/docker.png" /></a></div>
<h3>What Is Docker And Why Should You Care?</h3>
<p>Contrary to many others I believe that saying that Docker is a lightweight
virtual machine is a very good description. Another way to look at Docker is
<code>chroot</code> on steroids. The last explanation probably doesn't help much unless
you know what <code>chroot</code> is.</p>
<blockquote><pre><tt>Chroot is an operation that changes the apparent root directory for
the current running process and their children. A program that is run
in such a modified environment cannot access files and commands outside
that environmental directory tree. This modified environment is called
a chroot jail.
-- From Archwiki, chroot</tt></pre></blockquote>
<h3>VM vs. Docker</h3>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjmW-HjyB7spq8DopjztMlX_ct8eq3CM6PiFXAUeKom3oowUkjTe9U1uFbrVvDY7Ti7YTYqBw1H5LMOvvrRNifdIMpW2CLCbZYl6uiGGLLKGJdgd-XfrlU5NYTMw7m5aJyoBtCH/s1600/vm-vs-docker.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjmW-HjyB7spq8DopjztMlX_ct8eq3CM6PiFXAUeKom3oowUkjTe9U1uFbrVvDY7Ti7YTYqBw1H5LMOvvrRNifdIMpW2CLCbZYl6uiGGLLKGJdgd-XfrlU5NYTMw7m5aJyoBtCH/s320/vm-vs-docker.png" /></a></div>
<p>The image describes the difference between a VM and Docker. Instead of a
<em>hypervisor</em> with Guest OSes on top, Docker uses a <em>Docker engine</em> and
containers on top. Does this really tell us anything? What is the difference
between a "hypervisor" and the "Docker engine"? A nice way of illustrating this
difference is through listing the running processes on the Host.</p>
<p>The following simplified process trees illustrates the difference.</p>
<p>On the Host running the VM there is only <em>one</em> process running on the Host
even though there are many processes running in the VM.</p>
<blockquote><pre><tt><i><font color="#9A1900"># Running processes on Host for a VM</font></i>
$ pstree VM
-<font color="#990000">+=</font> /VirtualBox<font color="#990000">.</font>app
<font color="#990000">|</font>--<font color="#990000">=</font> coreos-vagrant
</tt></pre></blockquote>
<p>On the Host running the Docker Engine all the processes running are visible.
<em>The contained processes are running on the Host!</em> They can be inspected and
manipulated with normal commands like, <code>ps</code>, and <code>kill</code>.</p>
<blockquote><pre><tt><i><font color="#9A1900"># Running processes on Host for a Docker Engine</font></i>
$ pstree docker
-<font color="#990000">+=</font> /docker
<font color="#990000">|</font>--<font color="#990000">=</font> /bin/sh
<font color="#990000">|</font>--<font color="#990000">=</font> node server<font color="#990000">.</font>js
<font color="#990000">|</font>--<font color="#990000">=</font> go run app
<font color="#990000">|</font>--<font color="#990000">=</font> ruby server<font color="#990000">.</font>rb
<font color="#990000">...</font>
<font color="#990000">|</font>--<font color="#990000">=</font> /bin/bash
</tt></pre></blockquote>
<p>Now when everything is crystal clear, what does this mean? It means
that Docker containers are smaller, faster, and more easily integrated with
each other than VMs as the table illustrates.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgih1NLYekf7cC5yirW-G_His5r8VgOoE4aFl4LwFPQtohtpookBXRzDoPP_r6jvJdWD65EI2X3uTkYTSEbDhTqa03EnV87uE86sN1b3sfbC5ZeNCzEC75oARa1wlhISI54gEgl/s1600/vm-vs-docker-table.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgih1NLYekf7cC5yirW-G_His5r8VgOoE4aFl4LwFPQtohtpookBXRzDoPP_r6jvJdWD65EI2X3uTkYTSEbDhTqa03EnV87uE86sN1b3sfbC5ZeNCzEC75oARa1wlhISI54gEgl/s320/vm-vs-docker-table.png" /></a></div>
<p>The size of a small virtual machine image with Core OS is about 1.2 GB. The size
of a small container with busybox is 2.5 MB.</p>
<p>The startup time of a fast virtual machine is measured in minutes. The startup
time of a container is often less than a second.</p>
<p>Integrating virtual machines running on the same host must be done by setting
up the networking properly. Integrating containers is supported by Docker out
of the box.</p>
<p>So, containers are lightweight, fast and easily integrated, but that is not
all.</p>
<h3>Docker is a Contract</h3>
<p>Docker is also the contract between Developers and Operations. Developers and
Operations often have very different attitudes when it comes to choosing tools
and environments.</p>
<p>Developers want to use the next shiny thing, we want to use Node.js, Rust, Go,
Microservices, Cassandra, Hadoop, blablabla, blablabla, ...</p>
<p>Operations want to use the same as they used yesterday, what they used last
year, because it is proven, it works!</p>
<p>(Yes, I know this is stereotypical, but there is some truth in it :)</p>
<p>But, this is where Docker shines. Operations are satisfied because they only
have to care about one thing. They have to support deploying containers.
Developers are also happy. They can develop with whatever the fad of the day is
and then just stick it into a container and throw it over the wall to
Operations. Yippie ki-yay!</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9-BUV0B-A9ta5hgJyxv-obUkMci4-pDTb5ivkLtfz_TfgawVj-yMT7avG6nrS1OcAhdK0MPumikNC_Ob0HoakWDUrtg5sLsT0_2eLAp6AgX1Qy1Jjw25hp5l9noawAAL9lmqO/s1600/devs-loves-ops.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9-BUV0B-A9ta5hgJyxv-obUkMci4-pDTb5ivkLtfz_TfgawVj-yMT7avG6nrS1OcAhdK0MPumikNC_Ob0HoakWDUrtg5sLsT0_2eLAp6AgX1Qy1Jjw25hp5l9noawAAL9lmqO/s320/devs-loves-ops.png" /></a></div>
<p>But, it does not end here. Since Operations are, usually, better than
development when it comes to optimizing for production, they can help
developers build optimized containers that can be used for local development.
Not a bad situation at all.</p>
<h3>Better Utilization</h3>
<p>A few years ago, before virtualization, when we needed to create a new service,
we had to acquire an actual machine, hardware. It could take months, depending
on the processes of the company you were working for. One the server was in
place we created the service and most of the time it did not become the success
we were hoping for. The machine was ticking along with a CPU utilization of 5%.
Expensive!</p>
<p>Then, virtualization entered the arena and it was possible to spin up a new
machine in minutes. It was also possible to run multiple virtual machines on
the same hardware so the utilization increased from 5%. But, we still need to
have a virtual machine per service so the we cannot utilize the machine as much
as we would want.</p>
<p>Containerization is the next step in this process. Containers can be spun up in
seconds and they can be deployed at a much more granular level than virtual
machines.</p>
<h3>Dependencies</h3>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEimbZ0I4AUwFohvT2AKHJPbY0BkaM6GzTG6devbfDH_0XaAAPqMPvqg2YJl0ccs-Hgqz6OiwzuKi9YaSRq0FlNp5KOl_U4m_WUTlVWkOjRfUIHY-6di4xI2BGKqTHjT_twUbfwE/s1600/matrix-from-hell.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEimbZ0I4AUwFohvT2AKHJPbY0BkaM6GzTG6devbfDH_0XaAAPqMPvqg2YJl0ccs-Hgqz6OiwzuKi9YaSRq0FlNp5KOl_U4m_WUTlVWkOjRfUIHY-6di4xI2BGKqTHjT_twUbfwE/s320/matrix-from-hell.jpg" /></a></div>
<p>It is indeed nice that Docker can help us speed up our slow virtual machines
but why can't we just deploy all our services on the same machine? You already
know the answer, <em>dependency hell</em>. Installing multiple independent services on
a single machine, real or virtual, is a recipe for disaster. Docker Inc. calls
this the <em>matrix of hell</em>.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh6mh1iWon9sqOz93MEu9LfsZ6u_NPXKIJK8AaUnImEb9GdlXIO9FmGAen_r_DNkSFZUx7mQfp9OGGamKuFKr2vTUH_u2nWKVuuvnPFLeG4L4TgU6kOorfGoUJ1-b_w9FyBCb4N/s1600/matrix-from-hell-solution.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh6mh1iWon9sqOz93MEu9LfsZ6u_NPXKIJK8AaUnImEb9GdlXIO9FmGAen_r_DNkSFZUx7mQfp9OGGamKuFKr2vTUH_u2nWKVuuvnPFLeG4L4TgU6kOorfGoUJ1-b_w9FyBCb4N/s320/matrix-from-hell-solution.png" /></a></div>
<p>Docker eliminates the matrix of hell by keeping the dependencies contained inside
the containers.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiAQxz2Pfx36Rzat1xxGSk_RIpBCwa8_pUkA1y1L_zftZrbNsCRDKkbNnwd6KtCyrSDU7tcSDeqVrAvmx4OZANUSBMgPDLpE-prwCzpbKapZs2rKS7Z1cHr_GpBkBa8o4Muo5_2/s1600/roadrunner.gif" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiAQxz2Pfx36Rzat1xxGSk_RIpBCwa8_pUkA1y1L_zftZrbNsCRDKkbNnwd6KtCyrSDU7tcSDeqVrAvmx4OZANUSBMgPDLpE-prwCzpbKapZs2rKS7Z1cHr_GpBkBa8o4Muo5_2/s320/roadrunner.gif" /></a></div>
<h3>Speed</h3>
<p>Speed is of course always nice, but being 100 times faster is not only nice, it
changes what is possible. This much increase enables whole new possibilities.
It is now possible to create throw-away environments. Need to change your
entire development environment from Golang to Clojure? Fire up a container.
Need to provide a production database for integration and performance testing?
Fire up a container. Need to switch the entire production server from
Apache to Nginx? Fire up a container!</p>
<h2>How Does Docker Work?</h2>
<p>Docker is implemented as a client-server system; The Docker daemon runs on the
Host and it is accessed via a socket connection from the client. The client
may, but does not have to, be on the same machine as the daemon. The Docker CLI
client works the same way as any other client but it is usually connected
through a Unix domain socket instead of a TCP socket.</p>
<p>The daemon receives commands from the client and manages the containers on the
Host where it is running.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgmrEiZypRTlDukZIYeozGfOW7mZ1Fkd1Qt7ya8erMDgIAv7Q1yjU0_2U9awTsoNArUGcJ9ze2tgE29cPtn9iK7csiutRZTRUQy-J9E-xJQBLkEMRMKVE5e89AQnZDPtEgzgaMU/s1600/client-server.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgmrEiZypRTlDukZIYeozGfOW7mZ1Fkd1Qt7ya8erMDgIAv7Q1yjU0_2U9awTsoNArUGcJ9ze2tgE29cPtn9iK7csiutRZTRUQy-J9E-xJQBLkEMRMKVE5e89AQnZDPtEgzgaMU/s320/client-server.png" /></a></div>
<h3>Docker Concepts and Interactions</h3>
<ul>
<li>Host, the machine that is running the containers.</li>
<li>Image, a hierarchy of files, with meta-data for how to run a container.</li>
<li>Container, a contained running process, started from an image.</li>
<li>Registry, a repository of images.</li>
<li>Volume, storage outside the container.</li>
<li>Dockerfile, a script for creating images.</li>
</ul>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiIsJgX02vqGbxjp7TcnTM8iHPQNTe4cmYXB5c-fgWWnZi4_7m0FNE76Cbv4iVfGhe_g5aSwxrd-GOMDCvf_LstAFSHBcZMFyRC9D9RJ3L8AoG2fWbAzh8soJKFR1tJMYvq5a5G/s1600/docker-interactions.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiIsJgX02vqGbxjp7TcnTM8iHPQNTe4cmYXB5c-fgWWnZi4_7m0FNE76Cbv4iVfGhe_g5aSwxrd-GOMDCvf_LstAFSHBcZMFyRC9D9RJ3L8AoG2fWbAzh8soJKFR1tJMYvq5a5G/s320/docker-interactions.png" /></a></div>
<p>We can <code>build</code> an image from a <code>Dockerfile</code>. We can also create an image by
<code>commit</code>ing a running container. The image can be <code>tag</code>ged and it can be
<code>push</code>ed to and <code>pull</code>ed from a registry. A container is started by <code>run</code>ing or
<code>create</code>ing an image. A container can be <code>stop</code>ped and <code>start</code>ed. It can be
removed with <code>rm</code>.</p>
<h3>Images</h3>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrmJZHa0eBE4pvAL8WLQ038gLAzdw68hr3WeQVlcuJTjJdo2JF1YWdQELt9xC1Fsyhb-rjYQo2idtwEOx3IfkSSUD58R6T5uzn6BQP-FbeBmzq6cv1gfMhaFM4mBTIu9HHo8iU/s1600/docker-image.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrmJZHa0eBE4pvAL8WLQ038gLAzdw68hr3WeQVlcuJTjJdo2JF1YWdQELt9xC1Fsyhb-rjYQo2idtwEOx3IfkSSUD58R6T5uzn6BQP-FbeBmzq6cv1gfMhaFM4mBTIu9HHo8iU/s320/docker-image.png" /></a></div>
<p>An image is a file structure, with meta-data for how to run a container. The
image is built on a union filesystem, a filesystem built out of layers. Every
command in the <code>Dockerfile</code> creates a new layer in the filesystem.</p>
<p>When a container is started all images are merged together into what appears to the
process as unified. When files are removed in the union file system they are only marked
as deleted. The files will still exist in the layer where they were last present.</p>
<blockquote><pre><tt><i><font color="#9A1900"># Commands for interacting with images</font></i>
$ docker images <i><font color="#9A1900"># shows all images.</font></i>
$ docker <b><font color="#000080">import</font></b> <i><font color="#9A1900"># creates an image from a tarball.</font></i>
$ docker build <i><font color="#9A1900"># creates image from Dockerfile.</font></i>
$ docker commit <i><font color="#9A1900"># creates image from a container.</font></i>
$ docker rmi <i><font color="#9A1900"># removes an image.</font></i>
$ docker <b><font color="#0000FF">history</font></b> <i><font color="#9A1900"># list changes of an image.</font></i>
</tt></pre></blockquote>
<h4>Image Sizes</h4>
<p>Here are some data on commonly used images:</p>
<ul>
<li><code>scratch</code> - this is the ultimate base image and it has 0 files and 0 size.</li>
<li><code>busybox</code> - a minimal Unix weighing in at 2.5 MB and around 10000 files.</li>
<li><code>debian:jessie</code> - the latest Debian is 122 MB and around 18000 files.</li>
<li><code>ubuntu:14.04</code> - Ubuntu is 188 MB and has around 23000 files.</li>
</ul>
<h4>Creating images</h4>
<p>Images can be created with <code>docker commit container-id</code>, <code>docker import
url-to-tar</code>, or <code>docker build -f Dockerfile .</code></p>
<blockquote><pre><tt><i><font color="#9A1900"># Creating an image with commit</font></i>
$ docker run -i -t debian<font color="#990000">:</font>jessie bash
root@e6c7d21960<font color="#990000">:/</font><i><font color="#9A1900"># apt-get update</font></i>
root@e6c7d21960<font color="#990000">:/</font><i><font color="#9A1900"># apt-get install postgresql</font></i>
root@e6c7d21960<font color="#990000">:/</font><i><font color="#9A1900"># apt-get install node</font></i>
root@e6c7d21960<font color="#990000">:/</font><i><font color="#9A1900"># node --version</font></i>
root@e6c7d21960<font color="#990000">:/</font><i><font color="#9A1900"># curl https://iojs.org/dist/v1.2.0/iojs-v1.2.0-</font></i>
linux-x<font color="#993399">64</font><font color="#990000">.</font>tar<font color="#990000">.</font>gz -o iojs<font color="#990000">.</font>tgz
root@e6c7d21960<font color="#990000">:/</font><i><font color="#9A1900"># tar xzf iojs.tgz</font></i>
root@e6c7d21960<font color="#990000">:/</font><i><font color="#9A1900"># ls</font></i>
root@e6c7d21960<font color="#990000">:/</font><i><font color="#9A1900"># cd iojs-v1.2.0-linux-x64/</font></i>
root@e6c7d21960<font color="#990000">:/</font><i><font color="#9A1900"># ls</font></i>
root@e6c7d21960<font color="#990000">:/</font><i><font color="#9A1900"># cp -r * /usr/local/</font></i>
root@e6c7d21960<font color="#990000">:/</font><i><font color="#9A1900"># iojs --version</font></i>
<font color="#993399">1.2</font><font color="#990000">.</font><font color="#993399">0</font>
root@e6c7d21960<font color="#990000">:/</font><i><font color="#9A1900"># exit</font></i>
$ docker ps -l -q
e6c7d21960
$ docker commit e6c7d21960 postgres-iojs
daeb0b76283eac2e0c7f7504bdde2d49c721a1b03a50f750ea9982464cfccb1e
</tt></pre></blockquote>
<p>As you can see from the above session, it is possible to create images with
<code>docker commit</code> but it is kind messy and it is hard to reproduce. It is better
to create images with <code>Dockerfile</code>s since they are clear and are easily
reproduced.</p>
<blockquote><pre><tt>FROM debian<font color="#990000">:</font>jessie
<i><font color="#9A1900"># Dockerfile for postgres-iojs</font></i>
RUN apt<font color="#990000">-</font>get update
RUN apt<font color="#990000">-</font>get install postgresql
RUN curl https<font color="#990000">://</font>iojs<font color="#990000">.</font>org<font color="#990000">/</font>dist<font color="#990000">/</font>iojs<font color="#990000">-</font>v1<font color="#990000">.</font><font color="#993399">2.0</font><font color="#990000">.</font>tgz <font color="#990000">-</font>o iojs<font color="#990000">.</font>tgz
RUN tar xzf iojs<font color="#990000">.</font>tgz
RUN cp <font color="#990000">-</font>r iojs<font color="#990000">-</font>v1<font color="#990000">.</font><font color="#993399">2.0</font><font color="#990000">-</font>linux<font color="#990000">-</font>x64<font color="#990000">/*</font> <font color="#990000">/</font>usr<font color="#990000">/</font>local
</tt></pre></blockquote>
<p>Build it with</p>
<blockquote><pre><tt>$ docker build -tag postgres-iojs <font color="#990000">.</font>
</tt></pre></blockquote>
<p>Since every command in the <code>Dockerfile</code> creates a new layer it is often better
to run similar commands together. Group the commands with <code>and</code> and split them
over several lines for readability.</p>
<blockquote><pre><tt>FROM debian<font color="#990000">:</font>jessie
<i><font color="#9A1900"># Dockerfile for postgres-iojs</font></i>
RUN apt<font color="#990000">-</font>get update <font color="#990000">&&</font> <font color="#990000">\</font>
apt<font color="#990000">-</font>get install postgresql <font color="#990000">&&</font> <font color="#990000">\</font>
curl https<font color="#990000">://</font>iojs<font color="#990000">.</font>org<font color="#990000">/</font>dist<font color="#990000">/</font>iojs<font color="#990000">-</font>v1<font color="#990000">.</font><font color="#993399">2.0</font><font color="#990000">.</font>tgz <font color="#990000">-</font>o iojs<font color="#990000">.</font>tgz <font color="#990000">&&</font> <font color="#990000">\</font>
tar xzf iojs<font color="#990000">.</font>tgz <font color="#990000">&&</font> <font color="#990000">\</font>
cp <font color="#990000">-</font>r iojs<font color="#990000">-</font>v1<font color="#990000">.</font><font color="#993399">2.0</font><font color="#990000">-</font>linux<font color="#990000">-</font>x64<font color="#990000">/*</font> <font color="#990000">/</font>usr<font color="#990000">/</font>local
</tt></pre></blockquote>
<p>The ordering of the lines in the <code>Dockerfile</code> is important as Docker caches the
intermediate images, in order to speed up image building. Order your
<code>Dockerfile</code> by putting the lines that change more often at the bottom of the
file. ADD and COPY get special treatment from the cache and are re-run whenever
an affected file changes even though the line does not change.</p>
<h3>Dockerfile Commands</h3>
<p>The <code>Dockerfile</code> supports 13 commands. Some of the commands are used when
you build the image and some are used when you run a container from the image.
Here is a table of the commands and when they are used.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgDJFGgbyi1tWvstuXJ3dvtV62mK892aAFRrIghW6k44t-96UzeP_5mK6tPy1JTiKY0cGV5AUXj9qwCO27gTCmoTVOhVFa1Je6tpjJj3qho0xyPr84xVvj5dWjZ7LfdtPbZffqs/s1600/dockerfile-commands.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgDJFGgbyi1tWvstuXJ3dvtV62mK892aAFRrIghW6k44t-96UzeP_5mK6tPy1JTiKY0cGV5AUXj9qwCO27gTCmoTVOhVFa1Je6tpjJj3qho0xyPr84xVvj5dWjZ7LfdtPbZffqs/s320/dockerfile-commands.png" /></a></div>
<h4>BUILD Commands</h4>
<ul>
<li><code>FROM</code> - The image the new image will be based on.</li>
<li><code>MAINTAINER</code> - Name and email of the maintainer of this image.</li>
<li><code>COPY</code> - Copy a file or a directory into the image.</li>
<li><code>ADD</code> - Same as COPY, but handle URL:s and unpack tarballs automatically.</li>
<li><code>RUN</code> - Run a command inside the container, such as <code>apt-get install</code>.</li>
<li><code>ONBUILD</code> - Run commands when building an inherited Dockerfile.</li>
<li><code>.dockerignore</code> - Not a command, but it controls what files are added to the
build context. Should include <code>.git</code> and other files not needed when building
the image.</li>
</ul>
<h4>RUN Commands</h4>
<ul>
<li><code>CMD</code> - Default command to run when running the container. Can be overridden
with command line parameters.</li>
<li><code>ENV</code> - Set environment variable in the container.</li>
<li><code>EXPOSE</code> - Expose ports from the container. Must be explicitly exposed by the
run command to the Host with <code>-p</code> or <code>-P</code>.</li>
<li><code>VOLUME</code> - Specify that a directory should be stored outside the union file
system. If is not set with <code>docker run -v</code> it will be created in
<code>/var/lib/docker/volumes</code></li>
<li><code>ENTRYPOINT</code> - Specify a command that is not overridden by giving a new
command with <code>docker run image cmd</code>. It is mostly used to give a default
executable and use commands as parameters to it.</li>
</ul>
<h4>Both BUILD and RUN Commands</h4>
<ul>
<li><code>USER</code> - Set the user for RUN, CMD and ENTRYPOINT.</li>
<li><code>WORKDIR</code> - Sets the working directory for RUN, CMD, ENTRYPOINT, ADD and
COPY.</li>
</ul>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrmJZHa0eBE4pvAL8WLQ038gLAzdw68hr3WeQVlcuJTjJdo2JF1YWdQELt9xC1Fsyhb-rjYQo2idtwEOx3IfkSSUD58R6T5uzn6BQP-FbeBmzq6cv1gfMhaFM4mBTIu9HHo8iU/s1600/docker-image.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrmJZHa0eBE4pvAL8WLQ038gLAzdw68hr3WeQVlcuJTjJdo2JF1YWdQELt9xC1Fsyhb-rjYQo2idtwEOx3IfkSSUD58R6T5uzn6BQP-FbeBmzq6cv1gfMhaFM4mBTIu9HHo8iU/s320/docker-image.png" /></a></div>
<h2>Running Containers</h2>
<p>When a container is started, the process gets a new writable layer in the union
file system where it can execute.</p>
<p>Since version 1.5, it is also possible to
make this top layer read-only, forcing us to use volumes for all file output such
as logging, and temp-files.</p>
<blockquote><pre><tt><i><font color="#9A1900"># Commands for interacting with containers</font></i>
$ docker create <i><font color="#9A1900"># creates a container but does not start it.</font></i>
$ docker run <i><font color="#9A1900"># creates and starts a container.</font></i>
$ docker stop <i><font color="#9A1900"># stops it.</font></i>
$ docker start <i><font color="#9A1900"># will start it again.</font></i>
$ docker restart <i><font color="#9A1900"># restarts a container.</font></i>
$ docker rm <i><font color="#9A1900"># deletes a container.</font></i>
$ docker kill <i><font color="#9A1900"># sends a SIGKILL to a container.</font></i>
$ docker attach <i><font color="#9A1900"># will connect to a running container.</font></i>
$ docker <b><font color="#0000FF">wait</font></b> <i><font color="#9A1900"># blocks until container stops.</font></i>
$ docker <b><font color="#0000FF">exec</font></b> <i><font color="#9A1900"># executes a command in a running container.</font></i>
</tt></pre></blockquote>
<h3><code>docker run</code></h3>
<p>As the list above describes, <code>docker run</code> is the command used to start new
containers. Here are some common ways to run containers.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjVNOEqaDfgeSHIeBSjub9tAr8Wi4-h6qTSpREILAARqioJn-YzS8OMffvuhPEX6A49YWTetUMACDpgQv_Y3iHQNbhRN4wgkbtS_71KYjqoFP71sTluaf4C3F4ikDRRMdRHVj-9/s1600/container.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjVNOEqaDfgeSHIeBSjub9tAr8Wi4-h6qTSpREILAARqioJn-YzS8OMffvuhPEX6A49YWTetUMACDpgQv_Y3iHQNbhRN4wgkbtS_71KYjqoFP71sTluaf4C3F4ikDRRMdRHVj-9/s320/container.png" /></a></div>
<blockquote><pre><tt><i><font color="#9A1900"># Run a container interactively</font></i>
$ docker run -it --rm ubuntu
</tt></pre></blockquote>
<p>This is the way to run a container if you want to interact with it as a normal
terminal program. If you want to pipe into the container, you should not use the
<code>-t</code> option.</p>
<ul>
<li><code>--interactive (-i)</code> - send stdin to the process.</li>
<li><code>-tty (-t)</code> - tell the process that a terminal is present. This affects how the
process outputs data and how it treats signals such as (Ctrl-C).</li>
<li><code>--rm</code> - remove the container on exit.</li>
<li><code>ubuntu</code> - use the <code>ubuntu:latest</code> image.</li>
</ul>
<blockquote><pre><tt><i><font color="#9A1900"># Run a container in the background</font></i>
$ docker run -d hadoop
</tt></pre></blockquote>
<ul>
<li><code>--detached (-d)</code> - Run in detached mode, you can attach again with <code>docker
attach</code></li>
</ul>
<h3><code>docker run --env</code></h3>
<blockquote><pre><tt><i><font color="#9A1900"># Run a named container and pass it some environment variables</font></i>
$ docker run <font color="#990000">\</font>
--name mydb <font color="#990000">\</font>
--env <font color="#009900">MYSQL_USER</font><font color="#990000">=</font>db-user <font color="#990000">\</font>
-e <font color="#009900">MYSQL_PASSWORD</font><font color="#990000">=</font>secret <font color="#990000">\</font>
--env-file <font color="#990000">.</font>/mysql<font color="#990000">.</font>env <font color="#990000">\</font>
mysql
</tt></pre></blockquote>
<ul>
<li><code>--name</code> - name the container, otherwise it gets a random name.</li>
<li><code>-env (-e)</code> - Set the environment variable in the container</li>
<li><code>--env-file</code> - Set all environment variables in <code>env-file</code></li>
<li><code>mysql</code> - use the <code>mysql:latest</code> image.</li>
</ul>
<h3><code>docker run --publish</code></h3>
<blockquote><pre><tt><i><font color="#9A1900"># Publish container port 80 on a random port on the Host</font></i>
$ docker run -p <font color="#993399">80</font> nginx
<i><font color="#9A1900"># Publish container port 80 on port 8080 on the Host</font></i>
$ docker run -p <font color="#993399">8080</font><font color="#990000">:</font><font color="#993399">80</font> nginx
<i><font color="#9A1900"># Publish container port 80 on port 8080 on the localhost interface on the Host</font></i>
$ docker run -p <font color="#993399">127.0</font><font color="#990000">.</font><font color="#993399">0.1</font><font color="#990000">:</font><font color="#993399">8080</font><font color="#990000">:</font><font color="#993399">80</font> nginx
<i><font color="#9A1900"># Publish all EXPOSEd ports from the container on random ports on the Host</font></i>
$ docker run -P nginx
</tt></pre></blockquote>
<p>The nginx image, for example, exposes port 80 and 443.</p>
<blockquote><pre><tt> <font color="#993399">1</font> FROM debian<font color="#990000">:</font>wheezy
<font color="#993399">2</font>
<font color="#993399">3</font> MAINTAINER NGINX <font color="#FF0000">"docker-maint@nginx.com"</font>
<font color="#993399">21</font>
<font color="#993399">22</font> EXPOSE <font color="#993399">80</font> <font color="#993399">443</font>
<font color="#993399">23</font>
```
<i><font color="#9A1900">### docker run --link</font></i>
</tt></pre></blockquote>
<blockquote><pre><tt><i><font color="#9A1900"># Start a postgres container, named mydb</font></i>
$ docker run --name mydb postgres
<i><font color="#9A1900"># Link mydb as db into myqpp</font></i>
$ docker run --link mydb<font color="#990000">:</font>db myapp
</tt></pre></blockquote>
<p>Linking a container sets up networking from the linking container into the
linked container. It does two things:</p>
<ul>
<li>It updates the /etc/hosts with the link name given to the container, <code>db</code> in
the example above. Making it possible to access the container by the name
<code>db</code>. This is very good.</li>
<li>It creates environment variables for the EXPOSEd ports. This is practically
useless since I can access the same port by using a hostname:port combination
anyway.</li>
</ul>
<p>The linked networking is not constrained by the ports EXPOSEd by the image. All
ports are available to the linking container.</p>
<h3>docker run limits</h3>
<p>It is also possible to limit how much access the container has to the Host's
resources.</p>
<blockquote><pre><tt><i><font color="#9A1900"># Limit the amount of memory</font></i>
$ docker run -m 256m yourapp
<i><font color="#9A1900"># Limit the number of shares of the CPU this process uses (out of 1024)</font></i>
$ docker run --cpu-shares <font color="#993399">512</font> mypp
<i><font color="#9A1900"># Change the user for the process to www instead of root (good for security)</font></i>
$ docker run -u<font color="#990000">=</font>www nginx
</tt></pre></blockquote>
<p>Setting CPU shares to 512 out of 1024 does not mean that the process gets
access to half of the CPU, it means that it gets half as many shares as a
container that is run without any limit. If we have two containers running with
1024 shares and one with 512 shares the 512-container will get about 1 fifth
of the CPU shares.</p>
<h3>docker exec container</h3>
<p><code>docker exec</code> allows us to run commands inside already running containers.
This is very good for debugging among other things.</p>
<blockquote><pre><tt><i><font color="#9A1900"># Run a shell inside the container with id 6f2c42c0</font></i>
$ docker <b><font color="#0000FF">exec</font></b> -it 6f2c42c0 sh
</tt></pre></blockquote>
<h2>Volumes</h2>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhz2gzAS_kG7IyVXRXWST1CdcuQmYM4HuME17pjKSKmy5cCrGMOWEnH9KnL5Ty60kRkN5a7t76QHE3teZK-nd4jxvaVKg6N1SBHH_TysDSzues2SkAfE-9PLEZk9J0f09Kt8I2c/s1600/volumes.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhz2gzAS_kG7IyVXRXWST1CdcuQmYM4HuME17pjKSKmy5cCrGMOWEnH9KnL5Ty60kRkN5a7t76QHE3teZK-nd4jxvaVKg6N1SBHH_TysDSzues2SkAfE-9PLEZk9J0f09Kt8I2c/s320/volumes.png" /></a></div>
<p>Volumes provide persistent storage outside the container. That means the data
will not be saved if you commit the new image.</p>
<blockquote><pre><tt><i><font color="#9A1900"># Start a new nginx container with /var/log as a volume</font></i>
$ docker run -v /var/log nginx
</tt></pre></blockquote>
<p>Since the directory of the host is not given, the volume is created in
<code>/var/lib/docker/volumes/ec3c543bc..535</code>.</p>
<p>The exact name of the directory can be found by running <code>docker inspect
container-id</code>.</p>
<blockquote><pre><tt><i><font color="#9A1900"># Start a new nginx container with /var/log as a volume mapped to /tmp on Host</font></i>
$ docker run -v /tmp<font color="#990000">:</font>/var/log nginx
</tt></pre></blockquote>
<p>It is also possible to mount volumes from another container with
<code>--volumes-from</code>.</p>
<blockquote><pre><tt><i><font color="#9A1900"># Start a db container</font></i>
$ docker run -v /var/lib/postgresql/data --name mydb postgres
<i><font color="#9A1900"># Start a backup container with the volumes taken from the mydb container</font></i>
$ docker run --volumes-from mydb backup
</tt></pre></blockquote>
<h2>Docker Registries</h2>
<p>Docker Hub is the official repository for images. It supports public (free)
and private (fee) repositories. Repositories can be tagged as <em>official</em> and
this means that they are curated by the maintainers of the project (or someone
connected with it).</p>
<p>Docker Hub also supports automatic builds of projects hosted on Github and
Bitbucket. If automatic build is enabled an image will automatically be built
every time you push to your source code repository.</p>
<p>If you don't want to use automatic builds, you can also <code>docker push</code> directly
to Docker Hub. <code>docker pull</code> will pull images. <code>docker run</code> with an image that
does not exist locally will automatically initiate a <code>docker pull</code>.</p>
<p>It is also possible to host your images elsewhere. Docker maintains code for
<a href="https://github.com/docker/docker-registry">docker-registry</a> on Github. But, I
have found it to be slow and buggy.</p>
<p>Quay, Tutum, and Google also provides hosting of private docker images.</p>
<h2>Inspecting Containers</h2>
<p>A lot of commands are available for inspecting containers:</p>
<blockquote><pre><tt>$ docker ps <i><font color="#9A1900"># shows running containers.</font></i>
$ docker inspect <i><font color="#9A1900"># info on a container (incl. IP address).</font></i>
$ docker logs <i><font color="#9A1900"># gets logs from container.</font></i>
$ docker events <i><font color="#9A1900"># gets events from container.</font></i>
$ docker port <i><font color="#9A1900"># shows public facing port of container.</font></i>
$ docker top <i><font color="#9A1900"># shows running processes in container.</font></i>
$ docker diff <i><font color="#9A1900"># shows changed files in container's FS.</font></i>
$ docker stats <i><font color="#9A1900"># shows metrics, memory, cpu, filsystem</font></i>
</tt></pre></blockquote>
<p>I will only elaborate on <code>docker ps</code> and <code>docker inspect</code> since they are the
most important ones.</p>
<blockquote><pre><tt><i><font color="#9A1900"># List all containers, (--all means including stopped)</font></i>
$ docker ps --all
CONTAINER ID IMAGE COMMAND NAMES
9923ad197b65 busybox<font color="#990000">:</font>latest <font color="#FF0000">"sh"</font> romantic_fermat
fe7f682cf546 debian<font color="#990000">:</font>jessie <font color="#FF0000">"bash"</font> silly_bartik
09c707e2ec07 scratch<font color="#990000">:</font>latest <font color="#FF0000">"ls"</font> suspicious_perlman
b15c5c553202 mongo<font color="#990000">:</font><font color="#993399">2.6</font><font color="#990000">.</font><font color="#993399">7</font> <font color="#FF0000">"/entrypo some-mongo</font>
<font color="#FF0000">fbe1f24d7df8 busybox:latest "</font><b><font color="#0000FF">true</font></b><font color="#FF0000">" db_data</font>
<font color="#FF0000"># Inspect the container named silly_bartik</font>
<font color="#FF0000"># Output is shortened for brevity.</font>
<font color="#FF0000">$ docker inspect silly_bartik</font>
<font color="#FF0000"> 1 [{</font>
<font color="#FF0000"> 2 "</font>Args<font color="#FF0000">": [</font>
<font color="#FF0000"> 3 "</font>-c<font color="#FF0000">",</font>
<font color="#FF0000"> 4 "</font>/usr/local/bin/confd-watch<font color="#990000">.</font>sh<font color="#FF0000">"</font>
<font color="#FF0000"> 5 ],</font>
<font color="#FF0000"> 6 "</font>Config<font color="#FF0000">": {</font>
<font color="#FF0000"> 10 "</font>Hostname<font color="#FF0000">": "</font>3c012df7bab9<font color="#FF0000">",</font>
<font color="#FF0000"> 11 "</font>Image<font color="#FF0000">": "</font>andersjanmyr/nginx-confd<font color="#990000">:</font>development<font color="#FF0000">",</font>
<font color="#FF0000"> 12 },</font>
<font color="#FF0000"> 13 "</font>Id<font color="#FF0000">": "</font>3c012df7bab977a194199f1<font color="#FF0000">",</font>
<font color="#FF0000"> 14 "</font>Image<font color="#FF0000">": "</font>d3bd1f07cae1bd624e2e<font color="#FF0000">",</font>
<font color="#FF0000"> 15 "</font>NetworkSettings<font color="#FF0000">": {</font>
<font color="#FF0000"> 16 "</font>IPAddress<font color="#FF0000">": "",</font>
<font color="#FF0000"> 18 "</font>Ports<font color="#FF0000">": null</font>
<font color="#FF0000"> 19 },</font>
<font color="#FF0000"> 20 "</font>Volumes<font color="#FF0000">": {},</font>
<font color="#FF0000"> 22 }]</font>
</tt></pre></blockquote>
<h2>Tips and Tricks</h2>
<p>To get the id of a container is useful for scripting.</p>
<blockquote><pre><tt><i><font color="#9A1900"># Get the id (-q) of the last (-l) run container</font></i>
$ docker ps -l -q
c8044ab1a3d0
</tt></pre></blockquote>
<p><code>docker inspect</code> can take a format string, a Go template, and it allows you to
be more specific about what data you are interested in. Again, useful for
scripting.</p>
<blockquote><pre><tt>$ docker inspect -f <font color="#FF0000">'{{ .NetworkSettings.IPAddress }}'</font> 6f2c42c05500
<font color="#993399">172.17</font><font color="#990000">.</font><font color="#993399">0.11</font>
</tt></pre></blockquote>
<p>Use <code>docker exec</code> to interact with a running container.</p>
<blockquote><pre><tt><i><font color="#9A1900"># Get the environment variables of a running container.</font></i>
$ docker <b><font color="#0000FF">exec</font></b> -it 6f2c42c05500 env
<font color="#009900">PATH</font><font color="#990000">=</font>/usr/local/sbin<font color="#990000">:</font>/usr<font color="#990000">...</font>
<font color="#009900">HOSTNAME</font><font color="#990000">=</font>6f2c42c05500
<font color="#009900">REDIS_1_PORT</font><font color="#990000">=</font>tcp<font color="#990000">://</font><font color="#993399">172.17</font><font color="#990000">.</font><font color="#993399">0.9</font><font color="#990000">:</font><font color="#993399">6379</font>
<font color="#009900">REDIS_1_PORT_6379_TCP</font><font color="#990000">=</font>tcp<font color="#990000">://</font><font color="#993399">172.17</font><font color="#990000">.</font><font color="#993399">0.9</font><font color="#990000">:</font><font color="#993399">6379</font>
<font color="#990000">...</font>
</tt></pre></blockquote>
<p>Use volumes to avoid having to rebuild an image every time you run it. Every
time the below Dockerfile is built it copies the current directory into the
container.</p>
<blockquote><pre><tt> <font color="#993399">1</font> FROM dockerfile<font color="#990000">/</font>nodejs<font color="#990000">:</font>latest
<font color="#993399">2</font>
<font color="#993399">3</font> MAINTAINER Anders Janmyr <font color="#FF0000">"anders@janmyr.com"</font>
<font color="#993399">4</font> RUN apt<font color="#990000">-</font>get update <font color="#990000">&&</font> <font color="#990000">\</font>
<font color="#993399">5</font> apt<font color="#990000">-</font>get install zlib1g<font color="#990000">-</font>dev <font color="#990000">&&</font> <font color="#990000">\</font>
<font color="#993399">6</font> npm install <font color="#990000">-</font>g pm2 <font color="#990000">&&</font> <font color="#990000">\</font>
<font color="#993399">7</font> mkdir <font color="#990000">-</font>p <font color="#990000">/</font>srv<font color="#990000">/</font>app
<font color="#993399">8</font>
<font color="#993399">9</font> WORKDIR <font color="#990000">/</font>srv<font color="#990000">/</font>app
<font color="#993399">10</font> COPY <font color="#990000">.</font> <font color="#990000">/</font>srv<font color="#990000">/</font>app
<font color="#993399">11</font>
<font color="#993399">12</font> CMD pm2 start app<font color="#990000">.</font>js <font color="#990000">-</font>x <font color="#990000">-</font>i <font color="#993399">1</font> <font color="#990000">&&</font> pm2 logs
<font color="#993399">13</font>
</tt></pre></blockquote>
<blockquote><pre><tt><i><font color="#9A1900"># Build and run the image</font></i>
$ docker build -t myapp <font color="#990000">.</font>
$ docker run -it --rm myapp
</tt></pre></blockquote>
<p>To avoid the rebuild, build the image once and then mount the local directory
when you run it.</p>
<blockquote><pre><tt>$ docker run -it --rm -v <font color="#009900">$(PWD)</font><font color="#990000">:</font>/srv/app myapp
</tt></pre></blockquote>
<h2>Security</h2>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiyJYLptZbDWovlm199Ng1ROaDHm8u4e2nvYxsRwmv3qqBGgPbGhAlTmJUgdT8LC4Lv5D9pUtnNHfL4P9n6HkkSgUujj6CDnfHwuiT7bPNkFpTIZKpSYoWPM9NXlMTsblzolrdF/s1600/security.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiyJYLptZbDWovlm199Ng1ROaDHm8u4e2nvYxsRwmv3qqBGgPbGhAlTmJUgdT8LC4Lv5D9pUtnNHfL4P9n6HkkSgUujj6CDnfHwuiT7bPNkFpTIZKpSYoWPM9NXlMTsblzolrdF/s320/security.jpg" /></a></div>
<p>You may have heard that it is not secure to use Docker. This is not untrue, but
it does not have to be a problem.</p>
<p>The following security problems currently exists with Docker.</p>
<ul>
<li>Image signatures are not properly verified.</li>
<li>If you have <code>root</code> in a container you can, potentially, get root on the
entire box.</li>
</ul>
<h3>Security Remedies</h3>
<ul>
<li>Use trusted images from your private repositories.</li>
<li>Don't run containers as root, if possible.</li>
<li>Treat root in a container as root outside a container</li>
</ul>
<p>If you own all the containers running on the server, you don't have to worry
about them interacting with each other maliciously.</p>
<h2>Container "Options"</h2>
<p>I put "options" in quotes since there are not really any options at the moment,
but a lot of players want to get in the game. Ubuntu is working on something
called <code>LXD</code> and Microsoft on something called <code>Drawbridge</code>. But, the one that
seems most interesting is the one called <em>Rocket</em>.</p>
<p>Rocket is developed by Core OS, who is a big container (Docker) platform. The
reason for developing it is that they feel that Docker Inc. are bloating Docker
and also that they are moving into the same area as Core OS, which is container
hosting in the cloud.</p>
<p>With this new container specification they are trying to remove some of the
warts which Docker has for historical reasons and to provide a simple container
with support for <a href="http://0pointer.de/blog/projects/socket-activation.html">socket activation</a>
and security built in from the start.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjA3vN8mN2vKcxx2G7GQlz4WimVt02j1fzZBbbjwpY2i-SEu-y0Ibx9IawuRgITfQdBOE32Tvc0iCqJmODYQxDUfX4f3ILfsn94L6rcqpxGsPosnRhwXq6BkvyzGdxxwvhmDh2I/s1600/container-options.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjA3vN8mN2vKcxx2G7GQlz4WimVt02j1fzZBbbjwpY2i-SEu-y0Ibx9IawuRgITfQdBOE32Tvc0iCqJmODYQxDUfX4f3ILfsn94L6rcqpxGsPosnRhwXq6BkvyzGdxxwvhmDh2I/s320/container-options.png" /></a></div>
<h2>Orchestration</h2>
<p>When we split up our application into multiple different containers we get
some new problems. How do we make the different parts talk to each other?
On a single host? On multiple hosts?</p>
<p>Docker solves the problem of orchestration with on single host with links.</p>
<p>To simplify the linking of containers Docker provides a tool called
<code>docker-compose</code>. It was previously called <code>fig</code> and was developed by another
company which was recently acquired by Docker.</p>
<h3><code>docker-compose</code></h3>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj91ZPa5U8HU9OZp14x4jU492eLRjFPD2rw_lZN4JnnjXrLzd0fy78JD74c7sIMWKTlCBv0kB_ByoJbYptGupC_qv7oR8M-aA6tIE6p7-sgbshDyI-UlTVGBuBd10LsA7WUtJPL/s1600/fig.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj91ZPa5U8HU9OZp14x4jU492eLRjFPD2rw_lZN4JnnjXrLzd0fy78JD74c7sIMWKTlCBv0kB_ByoJbYptGupC_qv7oR8M-aA6tIE6p7-sgbshDyI-UlTVGBuBd10LsA7WUtJPL/s200/fig.png" /></a></div>
<p><code>docker-compose</code> declares the information for multiple containers in a single
file, <code>docker-compose.yml</code>. Here is an example of a file that manages two
containers, web and redis.</p>
<blockquote><pre><tt> <font color="#993399">1</font> web<font color="#990000">:</font>
<font color="#993399">2</font> build<font color="#990000">:</font> <font color="#990000">.</font>
<font color="#993399">3</font> command<font color="#990000">:</font> python app<font color="#990000">.</font>py
<font color="#993399">4</font> ports<font color="#990000">:</font>
<font color="#993399">5</font> <font color="#990000">-</font> <font color="#FF0000">"5000:5000"</font>
<font color="#993399">6</font> volumes<font color="#990000">:</font>
<font color="#993399">7</font> <font color="#990000">-</font> <font color="#990000">.:/</font>code
<font color="#993399">8</font> links<font color="#990000">:</font>
<font color="#993399">9</font> <font color="#990000">-</font> redis
<font color="#993399">10</font> redis<font color="#990000">:</font>
<font color="#993399">11</font> image<font color="#990000">:</font> redis
</tt></pre></blockquote>
<p>To start the above containers, you can run the command <code>docker-compose up</code>.</p>
<blockquote><pre><tt> <font color="#993399">1</font> $ docker-compose up
<font color="#993399">2</font> Pulling image orchardup/redis<font color="#990000">...</font>
<font color="#993399">3</font> Building web<font color="#990000">...</font>
<font color="#993399">4</font> Starting figtest_redis_1<font color="#990000">...</font>
<font color="#993399">5</font> Starting figtest_web_1<font color="#990000">...</font>
<font color="#993399">6</font> redis_1 <font color="#990000">|</font> <font color="#990000">[</font><font color="#993399">8</font><font color="#990000">]</font> <font color="#993399">02</font> Jan <font color="#993399">18</font><font color="#990000">:</font><font color="#993399">43</font><font color="#990000">:</font><font color="#993399">35.576</font> <i><font color="#9A1900"># Server</font></i>
<font color="#993399">7</font> started<font color="#990000">,</font> Redis version <font color="#993399">2.8</font><font color="#990000">.</font><font color="#993399">3</font>
<font color="#993399">8</font> web_1 <font color="#990000">|</font> <font color="#990000">*</font> Running on http<font color="#990000">://</font><font color="#993399">0.0</font><font color="#990000">.</font><font color="#993399">0.0</font><font color="#990000">:</font><font color="#993399">5000</font><font color="#990000">/</font>
</tt></pre></blockquote>
<p>It is also possible to start the containers in detached mode with
<code>docker-compose up -d</code> and I can find out what containers are running with
<code>docker-compose ps</code>.</p>
<blockquote><pre><tt> <font color="#993399">1</font> $ docker-compose up -d
<font color="#993399">2</font> Starting figtest_redis_1<font color="#990000">...</font>
<font color="#993399">3</font> Starting figtest_web_1<font color="#990000">...</font>
<font color="#993399">4</font> $ docker-compose ps
<font color="#993399">5</font> Name Command State Ports
<font color="#993399">6</font> ------------------------------------------------------------
<font color="#993399">7</font> figtest_redis_1 /usr/local/bin/run Up
<font color="#993399">8</font> figtest_web_1 /bin/sh -c python app<font color="#990000">.</font>py Up <font color="#993399">5000</font>-<font color="#990000">></font><font color="#993399">5000</font>
</tt></pre></blockquote>
<p>It is possible to run commands that work with a single container or commands
that work with all containers at once.</p>
<blockquote><pre><tt> <font color="#993399">1</font> <i><font color="#9A1900"># Get env variables for web container</font></i>
<font color="#993399">2</font> $ docker-compose run web env
<font color="#993399">3</font> <i><font color="#9A1900"># Scale to multiple containers</font></i>
<font color="#993399">4</font> $ docker-compose scale <font color="#009900">web</font><font color="#990000">=</font><font color="#993399">3</font> <font color="#009900">redis</font><font color="#990000">=</font><font color="#993399">2</font>
<font color="#993399">5</font> <i><font color="#9A1900"># Get logs for all containers</font></i>
<font color="#993399">6</font> $ docker-compose logs
</tt></pre></blockquote>
<p>As you can see from the above commands, scaling is supported. The application
must be written in a way that can handle multiple containers. Load-balancing is
not supported out of the box.</p>
<h2>Docker Hosting</h2>
<p>A number of companies want to get in on the business of hosting Docker in the
cloud. The image below shows a collection.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1FoKLs05oiWrTG9-0_uuN2ZeEqghbvkXNRm8jASdvzhyphenhyphen5o2QnasbKL3pCR1pAN73W205iNiOTHpKgoHCC6JUtV6j66kU9oB0Ylg5cFuMeAGzIZZlSD64WXJDKHtXTeuvP-0dW/s1600/docker-hosting-providers.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1FoKLs05oiWrTG9-0_uuN2ZeEqghbvkXNRm8jASdvzhyphenhyphen5o2QnasbKL3pCR1pAN73W205iNiOTHpKgoHCC6JUtV6j66kU9oB0Ylg5cFuMeAGzIZZlSD64WXJDKHtXTeuvP-0dW/s320/docker-hosting-providers.png" /></a></div>
<p>These provider try to solve different problems, from simple hosting to becoming
a "cloud operating system". I will only elaborate on two of them</p>
<h3>Core OS</h3>
<p>As the image shows, Core OS is a collection of services to enable hosting of
multiple containers in a Core OS cluster.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiYgMfbF3fvjcXOy2wNTSHR4I0ePg05Zmyk3BYKhSQTmbZp2QidjcCPNiq2yBT2xaWslsz-jSaQNm9E7u29RfmDrTrTL_UB4nCAxUeW3PxR27l6-frouqYAv9QoMkwabJyNil3j/s1600/core-os.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiYgMfbF3fvjcXOy2wNTSHR4I0ePg05Zmyk3BYKhSQTmbZp2QidjcCPNiq2yBT2xaWslsz-jSaQNm9E7u29RfmDrTrTL_UB4nCAxUeW3PxR27l6-frouqYAv9QoMkwabJyNil3j/s320/core-os.png" /></a></div>
<ul>
<li>The Core OS Linux distribution is a stripped down Linux. It uses 114MB of RAM
on initial boot. It does not provide a package manager, since it uses Docker
or their own Rocket container to run everything.</li>
<li>Core OS uses Docker (or Rocket) to install an application on a host.</li>
<li>It uses <code>systemd</code> as init-service since it has great performance, handles
start-up dependencies well, has great logging, and supports socket-activation.</li>
<li><code>etcd</code> is a distributed, consistent key value store for shared configuration
and service discovery.</li>
<li><code>fleet</code> is a cluster manager. It is an extension of <code>systemd</code> to work with
multiple machines. It uses <code>etcd</code> to manage configuration and it is running
on every Core OS machine.</li>
</ul>
<h3>AWS</h3>
<p>It is possible to host Docker containers on Amazon in two ways.</p>
<ul>
<li>Elastic Beanstalk can deploy Docker containers. This works fine but I find it
to be very slow. A new deploy takes several minutes and it does not feel
right when a container can be started in seconds.</li>
<li>ECS, Elastic Container Service, is Amazon's upcoming container cluster
solution. It is currently in preview 3 and it looks very promising. Just as
with Amazon's other services, you interact with it through simple web service
calls.</li>
</ul>
<h2>Summary</h2>
<ul>
<li>Docker is here to stay.</li>
<li>It fixes dependency hell.</li>
<li>Containers are fast!</li>
<li>Cluster solutions exists, but don't expect them to be seamless, yet!</li>
</ul>
Anders Janmyrhttp://www.blogger.com/profile/16081537018884349622noreply@blogger.com0tag:blogger.com,1999:blog-34049130.post-32364841050523682112014-12-16T16:22:00.001+01:002014-12-16T16:22:54.167+01:00Lambda, Javascript Micro-Services on AWS<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjByZYXx4-RZMve92q7RHlx8y9xlW2yIssd-3gGb5UsiZ5HWnIUfjOPQh4tAo8oHOP1h01j2v5BaGLQQmqU7vCXXogRErtkpnRBfY2FNeUUcqGQtcJp9SFzbu28R_HEofsjfxw8/s1600/Lambda.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjByZYXx4-RZMve92q7RHlx8y9xlW2yIssd-3gGb5UsiZ5HWnIUfjOPQh4tAo8oHOP1h01j2v5BaGLQQmqU7vCXXogRErtkpnRBfY2FNeUUcqGQtcJp9SFzbu28R_HEofsjfxw8/s320/Lambda.png" /></a></div>
<p>Amazon just released a bunch of new services. My favorite is Lambda.
Lambda allows me to deploy simple micro-services without having to setup any
servers at all. Everything is hosted in the AWS cloud. Another cool thing
about Lambda services is that the default runtime is Node.js!</p>
<p>To get access to AWS Lambda, you have to sign in to the [AWS Console] and
select the <code>Lambda</code> service. You have to fill out a form to request access,
which may take a while to come through. Once you have access you can edit the
functions in a web form.</p>
<p>A lambda service is a Node module which exports an object with one function,
the handler. In the AWS examples this is usually called <em>handler</em> and I'm going
to follow their example.</p>
<p>Here is a simple function that can be edited and invoked in the online <em>Lambda
Edit/Test</em> tool.</p>
<blockquote><pre><tt><i><font color="#9A1900">// hello-event.js</font></i>
exports<font color="#990000">.</font>handler <font color="#990000">=</font> <b><font color="#0000FF">function</font></b><font color="#990000">(</font>event<font color="#990000">,</font> context<font color="#990000">)</font> <font color="#FF0000">{</font>
console<font color="#990000">.</font><b><font color="#000000">log</font></b><font color="#990000">(</font><font color="#FF0000">'Hello'</font><font color="#990000">,</font> event<font color="#990000">);</font>
context<font color="#990000">.</font><b><font color="#000000">done</font></b><font color="#990000">(</font><b><font color="#0000FF">null</font></b><font color="#990000">,</font> <font color="#FF0000">'Success'</font><font color="#990000">);</font>
<font color="#FF0000">}</font>
</tt></pre></blockquote>
<p>The event is any JSON object and since a String is a valid object it can be
invoked with <code>&quot;Tapir&quot;</code>, which results in the following output in Lambda tool.</p>
<blockquote><pre><tt>Logs
----
START RequestId<font color="#990000">:</font> 3e21d80e-<font color="#993399">7e31</font>-<font color="#993399">11e4</font>-912c-2f870de05098
<font color="#993399">2014</font>-<font color="#993399">12</font>-07T16<font color="#990000">:</font><font color="#993399">51</font><font color="#990000">:</font><font color="#993399">47</font><font color="#990000">.</font>163Z 3e21d80e-<font color="#993399">7e31</font>-<font color="#993399">11e4</font>-912c-2f870de05098 Hello Tapir
END RequestId<font color="#990000">:</font> 3e21d80e-<font color="#993399">7e31</font>-<font color="#993399">11e4</font>-912c-2f870de05098
REPORT RequestId<font color="#990000">:</font> 3e21d80e-<font color="#993399">7e31</font>-<font color="#993399">11e4</font>-912c-2f870de05098 Duration<font color="#990000">:</font> <font color="#993399">3.89</font> ms Billed Duration<font color="#990000">:</font> <font color="#993399">100</font> ms Memory Size<font color="#990000">:</font> <font color="#993399">128</font> MB Max Memory Used<font color="#990000">:</font> <font color="#993399">9</font> MB
Message
-------
Success
</tt></pre></blockquote>
<p>Working in the Lambda online tool is sufficient for simple examples examples
but quickly gets annoying and once you need to add extra modules, you have to
upload zip-archives and this is both error prone and tedious. Here is a simple
script to zip relevant files and upload them to Lambda. Make sure to update the
region and the role to your own specific properties.</p>
<blockquote><pre><tt><i><font color="#9A1900">#!/bin/bash</font></i>
<i><font color="#9A1900">#</font></i>
<i><font color="#9A1900"># upload-lambda.sh</font></i>
<i><font color="#9A1900"># Zip and upload lambda function</font></i>
<i><font color="#9A1900">#</font></i>
<font color="#009900">program</font><font color="#990000">=</font>`basename <font color="#009900">$0</font>`
<b><font color="#0000FF">set</font></b> -o errexit
<b><font color="#000000">function usage()</font></b> {
echo <font color="#FF0000">"Usage: $program <function.js>"</font>
}
<b><font color="#0000FF">if</font></b> <font color="#990000">[</font> <font color="#009900">$#</font> -lt <font color="#993399">1</font> <font color="#990000">]</font>
<b><font color="#0000FF">then</font></b>
echo <font color="#FF0000">'Missing required parameters'</font>
usage
<b><font color="#0000FF">exit</font></b> <font color="#993399">1</font>
<b><font color="#0000FF">fi</font></b>
<font color="#009900">main</font><font color="#990000">=</font><font color="#009900">${1%.js}</font>
<font color="#009900">file</font><font color="#990000">=</font><font color="#FF0000">"./${main}.js"</font>
<font color="#009900">zip</font><font color="#990000">=</font><font color="#FF0000">"./${main}.zip"</font>
<font color="#009900">role</font><font color="#990000">=</font><font color="#FF0000">'arn:aws:iam::638281126589:role/lambda_exec_role'</font>
<font color="#009900">region</font><font color="#990000">=</font><font color="#FF0000">'eu-west-1'</font>
<b><font color="#000000">zip_package()</font></b> {
zip -r <font color="#009900">$zip</font> <font color="#009900">$file</font> lib node_modules
}
<b><font color="#000000">upload_package()</font></b> {
aws lambda upload-function <font color="#990000">\</font>
--region <font color="#009900">$region</font> <font color="#990000">\</font>
--role <font color="#009900">$role</font><font color="#990000">\</font>
--function-name <font color="#009900">$main</font> <font color="#990000">\</font>
--function-zip <font color="#009900">$zip</font> <font color="#990000">\</font>
--mode event <font color="#990000">\</font>
--handler <font color="#009900">$main</font><font color="#990000">.</font>handler <font color="#990000">\</font>
--runtime nodejs <font color="#990000">\</font>
--debug <font color="#990000">\</font>
--timeout <font color="#993399">10</font> <font color="#990000">\</font>
--memory-size <font color="#993399">128</font>
}
<i><font color="#9A1900"># main</font></i>
zip_package
upload_package
</tt></pre></blockquote>
<h2>A Larger Example</h2>
<p>Now that I know the Lambda works it is time to try out something more
elaborate. I have read that it is not only possible to get access to npm
modules but I also have access to the operating system when writing my
service.</p>
<p>My bigger example consists of something I often have use for, a way to serve
media files so that I don't have to check them into git. The way I want to do
this is to upload a tarball to S3 and then have Lambda unpack the archive,
checksum the files and upload them into another bucket.</p>
<p>Something like this:</p>
<ul>
<li>React to the <code>ObjectCreated:Put</code> event</li>
<li>Download the tarball from S3</li>
<li>Extract tarball into temp directory</li>
<li>Checksum the files and rename them with the checksum</li>
<li>Upload the checksummed file to another S3 bucket</li>
<li>Upload an index of the files with mapping from old to new filename.</li>
</ul>
<h2>React to <code>ObjectCreated:Put</code> event</h2>
<p>An AWS S3 <code>ObjectCreated:Put</code> event looks something like this in a trimmed
down format</p>
<blockquote><pre><tt><font color="#FF0000">{</font>
<font color="#FF0000">"Records"</font><font color="#990000">:</font> <font color="#990000">[</font> <font color="#FF0000">{</font>
<font color="#FF0000">"eventVersion"</font><font color="#990000">:</font> <font color="#FF0000">"2.0"</font><font color="#990000">,</font>
<font color="#FF0000">"eventSource"</font><font color="#990000">:</font> <font color="#FF0000">"aws:s3"</font><font color="#990000">,</font>
<font color="#FF0000">"eventName"</font><font color="#990000">:</font> <font color="#FF0000">"ObjectCreated:Put"</font><font color="#990000">,</font>
<font color="#FF0000">"s3"</font><font color="#990000">:</font> <font color="#FF0000">{</font>
<font color="#FF0000">"bucket"</font><font color="#990000">:</font> <font color="#FF0000">{</font>
<font color="#FF0000">"name"</font><font color="#990000">:</font> <font color="#FF0000">"anders-source"</font><font color="#990000">,</font>
<font color="#FF0000">}</font><font color="#990000">,</font>
<font color="#FF0000">"object"</font><font color="#990000">:</font> <font color="#FF0000">{</font>
<font color="#FF0000">"key"</font><font color="#990000">:</font> <font color="#FF0000">"tapirs.tgz"</font><font color="#990000">,</font>
<font color="#FF0000">"size"</font><font color="#990000">:</font> <font color="#993399">1024</font><font color="#990000">,</font>
<font color="#FF0000">"eTag"</font><font color="#990000">:</font> <font color="#FF0000">"d41d8cd98f00b204e9800998ecf8427e"</font>
<font color="#FF0000">}</font>
<font color="#FF0000">}</font>
<font color="#FF0000">}</font>
<font color="#990000">]</font>
<font color="#FF0000">}</font>
</tt></pre></blockquote>
<p>To handle this event we need a handler function. All the handler needs to do is
to extract the relevant properties from the file and then call <code>assetify</code> which
will do the rest of the work. Breaking up the code like this allows me to use
<code>assetify</code> locally and not only as a Lambda handler.</p>
<blockquote><pre><tt>assetify<font color="#990000">.</font>handler <font color="#990000">=</font> function<font color="#990000">(</font>event<font color="#990000">,</font> context<font color="#990000">)</font> {
console<font color="#990000">.</font>log<font color="#990000">(</font><font color="#FF0000">'Received event:'</font><font color="#990000">);</font>
console<font color="#990000">.</font>log<font color="#990000">(</font>JSON<font color="#990000">.</font>stringify<font color="#990000">(</font>event<font color="#990000">,</font> null<font color="#990000">,</font> <font color="#FF0000">' '</font><font color="#990000">));</font>
var bucket <font color="#990000">=</font> event<font color="#990000">.</font>Records<font color="#990000">[</font><font color="#993399">0</font><font color="#990000">].</font>s3<font color="#990000">.</font>bucket<font color="#990000">.</font>name<font color="#990000">;</font>
var key <font color="#990000">=</font> event<font color="#990000">.</font>Records<font color="#990000">[</font><font color="#993399">0</font><font color="#990000">].</font>s3<font color="#990000">.</font>object<font color="#990000">.</font>key<font color="#990000">;</font>
assetify<font color="#990000">(</font>bucket<font color="#990000">,</font> key<font color="#990000">,</font> function<font color="#990000">(</font>err<font color="#990000">,</font> result<font color="#990000">)</font> {
context<font color="#990000">.</font><b><font color="#0000FF">done</font></b><font color="#990000">(</font>err<font color="#990000">,</font> util<font color="#990000">.</font>inspect<font color="#990000">(</font>result<font color="#990000">));</font>
}<font color="#990000">);</font>
}<font color="#990000">;</font>
</tt></pre></blockquote>
<h2><code>assetify</code></h2>
<p>In order to use <code>assetify</code> as a normal module on a local machine I export the
function with <code>module.exports</code>. This code needs to come before the
<code>assetify.handler</code> declaration above. When exported this way, it is possible to
require the function without involving Lambda.</p>
<blockquote><pre><tt><b><font color="#0000FF">function</font></b> <b><font color="#000000">assetify</font></b><font color="#990000">(</font>sourceBucket<font color="#990000">,</font> key<font color="#990000">,</font> callback<font color="#990000">)</font> <font color="#FF0000">{</font>
<b><font color="#0000FF">var</font></b> tgzRegex <font color="#990000">=</font> <b><font color="#0000FF">new</font></b> <b><font color="#000000">RegExp</font></b><font color="#990000">(</font><font color="#FF0000">'</font><font color="#CC33CC">\\</font><font color="#FF0000">.tgz'</font><font color="#990000">);</font>
<b><font color="#0000FF">if</font></b> <font color="#990000">(!</font>key<font color="#990000">.</font><b><font color="#000000">match</font></b><font color="#990000">(</font>tgzRegex<font color="#990000">))</font> <b><font color="#0000FF">return</font></b> <b><font color="#000000">callback</font></b><font color="#990000">(</font><font color="#FF0000">'no match'</font><font color="#990000">);</font>
<b><font color="#0000FF">var</font></b> prefix <font color="#990000">=</font> path<font color="#990000">.</font><b><font color="#000000">basename</font></b><font color="#990000">(</font>key<font color="#990000">,</font> <font color="#FF0000">'.tgz'</font><font color="#990000">);</font>
async<font color="#990000">.</font><b><font color="#000000">waterfall</font></b><font color="#990000">([</font>
downloadFile<font color="#990000">.</font><b><font color="#000000">bind</font></b><font color="#990000">(</font><b><font color="#0000FF">null</font></b><font color="#990000">,</font> sourceBucket<font color="#990000">,</font> key<font color="#990000">),</font>
extractTarBall<font color="#990000">,</font>
checksumFiles<font color="#990000">,</font>
uploadFiles<font color="#990000">.</font><b><font color="#000000">bind</font></b><font color="#990000">(</font><b><font color="#0000FF">null</font></b><font color="#990000">,</font> prefix<font color="#990000">),</font>
uploadIndex<font color="#990000">.</font><b><font color="#000000">bind</font></b><font color="#990000">(</font><b><font color="#0000FF">null</font></b><font color="#990000">,</font> prefix<font color="#990000">)</font>
<font color="#990000">],</font> <b><font color="#0000FF">function</font></b><font color="#990000">(</font>err<font color="#990000">,</font> result<font color="#990000">)</font> <font color="#FF0000">{</font>
<b><font color="#0000FF">if</font></b> <font color="#990000">(</font>err<font color="#990000">)</font> <b><font color="#0000FF">return</font></b> <b><font color="#000000">callback</font></b><font color="#990000">(</font>err<font color="#990000">);</font>
<b><font color="#000000">callback</font></b><font color="#990000">(</font><b><font color="#0000FF">null</font></b><font color="#990000">,</font> result<font color="#990000">);</font>
<font color="#FF0000">}</font><font color="#990000">);</font>
<font color="#FF0000">}</font>
module<font color="#990000">.</font>exports <font color="#990000">=</font> assetify<font color="#990000">;</font>
</tt></pre></blockquote>
<p>I'm using <code>async.waterfall</code> in combination with <code>bind</code> to get a nice flat
structure of the code which clearly resembles the described flow above.</p>
<h2>Download file</h2>
<p>The <code>downloadFile</code> function uses a nice feature of <code>s3.getObject</code>, streaming.
After creating a temporary file with <code>tmp.file</code>, I create a request and then I
stream the contents from S3 directly into a write stream. Very nice! I also
need to hook up some event handler to allow me to notify the callback once the
streaming is complete.</p>
<blockquote><pre><tt><b><font color="#0000FF">function</font></b> <b><font color="#000000">downloadFile</font></b><font color="#990000">(</font>sourceBucket<font color="#990000">,</font> key<font color="#990000">,</font> callback<font color="#990000">)</font> <font color="#FF0000">{</font>
console<font color="#990000">.</font><b><font color="#000000">log</font></b><font color="#990000">(</font><font color="#FF0000">'downloadFile'</font><font color="#990000">,</font> sourceBucket<font color="#990000">,</font> key<font color="#990000">)</font>
tmp<font color="#990000">.</font><b><font color="#000000">file</font></b><font color="#990000">(</font><font color="#FF0000">{</font>postfix<font color="#990000">:</font> <font color="#FF0000">'.tgz'</font><font color="#FF0000">}</font><font color="#990000">,</font> <b><font color="#0000FF">function</font></b> <b><font color="#000000">tmpCreated</font></b><font color="#990000">(</font>err<font color="#990000">,</font> tmpfile<font color="#990000">)</font> <font color="#FF0000">{</font>
<b><font color="#0000FF">if</font></b> <font color="#990000">(</font>err<font color="#990000">)</font> <b><font color="#0000FF">return</font></b> <b><font color="#000000">callback</font></b><font color="#990000">(</font>err<font color="#990000">);</font>
<b><font color="#0000FF">var</font></b> awsRequest <font color="#990000">=</font> s3<font color="#990000">.</font><b><font color="#000000">getObject</font></b><font color="#990000">(</font><font color="#FF0000">{</font>Bucket<font color="#990000">:</font> sourceBucket<font color="#990000">,</font> Key<font color="#990000">:</font>key<font color="#FF0000">}</font><font color="#990000">);</font>
awsRequest<font color="#990000">.</font><b><font color="#000000">on</font></b><font color="#990000">(</font><font color="#FF0000">'success'</font><font color="#990000">,</font> <b><font color="#0000FF">function</font></b><font color="#990000">()</font> <font color="#FF0000">{</font>
<b><font color="#0000FF">return</font></b> <b><font color="#000000">callback</font></b><font color="#990000">(</font><b><font color="#0000FF">null</font></b><font color="#990000">,</font> tmpfile<font color="#990000">);</font>
<font color="#FF0000">}</font><font color="#990000">);</font>
awsRequest<font color="#990000">.</font><b><font color="#000000">on</font></b><font color="#990000">(</font><font color="#FF0000">'error'</font><font color="#990000">,</font> <b><font color="#0000FF">function</font></b><font color="#990000">(</font>response<font color="#990000">)</font> <font color="#FF0000">{</font>
<b><font color="#0000FF">return</font></b> <b><font color="#000000">callback</font></b><font color="#990000">(</font>response<font color="#990000">.</font>error<font color="#990000">);</font>
<font color="#FF0000">}</font><font color="#990000">);</font>
<b><font color="#0000FF">var</font></b> stream <font color="#990000">=</font> fs<font color="#990000">.</font><b><font color="#000000">createWriteStream</font></b><font color="#990000">(</font>tmpfile<font color="#990000">);</font>
awsRequest<font color="#990000">.</font><b><font color="#000000">createReadStream</font></b><font color="#990000">().</font><b><font color="#000000">pipe</font></b><font color="#990000">(</font>stream<font color="#990000">);</font>
<font color="#FF0000">}</font><font color="#990000">);</font>
<font color="#FF0000">}</font>
</tt></pre></blockquote>
<h2>Extract tarball</h2>
<p>In order to extract the tarball I'm using the ordinary <code>tar</code> command instead of
relying on a Node module. This works fine as Lambda seems to include a full
standard AWS distribution. Very nice to have access to all the common Unix
utilities. The <code>glob</code> function makes it easy to traverse the full tree
structure of the archive and I use this to return (or pass on via callback) a
map of filenames to the temporary files.</p>
<blockquote><pre><tt><b><font color="#0000FF">function</font></b> <b><font color="#000000">extractTarBall</font></b><font color="#990000">(</font>tarfile<font color="#990000">,</font> callback<font color="#990000">)</font> <font color="#FF0000">{</font>
tmp<font color="#990000">.</font><b><font color="#000000">dir</font></b><font color="#990000">(</font><b><font color="#0000FF">function</font></b><font color="#990000">(</font>err<font color="#990000">,</font> dir<font color="#990000">)</font> <font color="#FF0000">{</font>
<b><font color="#0000FF">if</font></b> <font color="#990000">(</font>err<font color="#990000">)</font> <b><font color="#0000FF">return</font></b> <b><font color="#000000">callback</font></b><font color="#990000">(</font>err<font color="#990000">);</font>
<b><font color="#0000FF">var</font></b> cmd <font color="#990000">=</font> <font color="#FF0000">'tar -xzf '</font> <font color="#990000">+</font> tarfile <font color="#990000">+</font> <font color="#FF0000">' -C '</font> <font color="#990000">+</font> dir<font color="#990000">;</font>
<b><font color="#000000">exec</font></b><font color="#990000">(</font>cmd<font color="#990000">,</font> <b><font color="#0000FF">function</font></b> <font color="#990000">(</font>err<font color="#990000">)</font> <font color="#FF0000">{</font>
<b><font color="#0000FF">if</font></b> <font color="#990000">(</font>err<font color="#990000">)</font> <b><font color="#0000FF">return</font></b> <b><font color="#000000">callback</font></b><font color="#990000">(</font>err<font color="#990000">);</font>
<b><font color="#000000">glob</font></b><font color="#990000">(</font>dir <font color="#990000">+</font> <font color="#FF0000">'**/*.*'</font><font color="#990000">,</font> <b><font color="#0000FF">function</font></b><font color="#990000">(</font>err<font color="#990000">,</font> files<font color="#990000">)</font> <font color="#FF0000">{</font>
<b><font color="#0000FF">if</font></b> <font color="#990000">(</font>err<font color="#990000">)</font> <b><font color="#0000FF">return</font></b> <b><font color="#000000">callback</font></b><font color="#990000">(</font>err<font color="#990000">);</font>
<b><font color="#0000FF">var</font></b> fs <font color="#990000">=</font> files<font color="#990000">.</font><b><font color="#000000">map</font></b><font color="#990000">(</font><b><font color="#0000FF">function</font></b><font color="#990000">(</font>file<font color="#990000">)</font> <font color="#FF0000">{</font>
<b><font color="#0000FF">return</font></b> <font color="#FF0000">{</font>
path<font color="#990000">:</font> file<font color="#990000">,</font>
originalFile<font color="#990000">:</font> file<font color="#990000">.</font><b><font color="#000000">replace</font></b><font color="#990000">(</font>dir<font color="#990000">,</font> <font color="#FF0000">''</font><font color="#990000">)</font>
<font color="#FF0000">}</font><font color="#990000">;</font>
<font color="#FF0000">}</font><font color="#990000">);</font>
<b><font color="#0000FF">return</font></b> <b><font color="#000000">callback</font></b><font color="#990000">(</font><b><font color="#0000FF">null</font></b><font color="#990000">,</font> fs<font color="#990000">);</font>
<font color="#FF0000">}</font><font color="#990000">);</font>
<font color="#FF0000">}</font><font color="#990000">);</font>
<font color="#FF0000">}</font><font color="#990000">);</font>
<font color="#FF0000">}</font>
</tt></pre></blockquote>
<h2>Checksum</h2>
<p><code>checksumFiles</code> uses <code>async.map</code> to call the singular version <code>checksumFile</code>.
This creates a checksum of the file and does some string manipulation in order
to create a name with a checksum in it.</p>
<blockquote><pre><tt><b><font color="#0000FF">function</font></b> <b><font color="#000000">checksumFiles</font></b><font color="#990000">(</font>files<font color="#990000">,</font> callback<font color="#990000">)</font> <font color="#FF0000">{</font>
async<font color="#990000">.</font><b><font color="#000000">map</font></b><font color="#990000">(</font>files<font color="#990000">,</font> checksumFile<font color="#990000">,</font> callback<font color="#990000">);</font>
<font color="#FF0000">}</font>
<b><font color="#0000FF">function</font></b> <b><font color="#000000">checksumFile</font></b><font color="#990000">(</font>file<font color="#990000">,</font> callback<font color="#990000">)</font> <font color="#FF0000">{</font>
checksum<font color="#990000">.</font><b><font color="#000000">file</font></b><font color="#990000">(</font>file<font color="#990000">.</font>path<font color="#990000">,</font> <font color="#FF0000">{</font> algorithm<font color="#990000">:</font> <font color="#FF0000">'md5'</font><font color="#FF0000">}</font><font color="#990000">,</font> <b><font color="#0000FF">function</font></b><font color="#990000">(</font>err<font color="#990000">,</font> sum<font color="#990000">)</font> <font color="#FF0000">{</font>
<b><font color="#0000FF">if</font></b> <font color="#990000">(</font>err<font color="#990000">)</font> <b><font color="#0000FF">return</font></b> <b><font color="#000000">callback</font></b><font color="#990000">(</font>err<font color="#990000">);</font>
<b><font color="#0000FF">var</font></b> filename <font color="#990000">=</font> file<font color="#990000">.</font>originalFile<font color="#990000">;</font>
<b><font color="#0000FF">var</font></b> ext <font color="#990000">=</font> path<font color="#990000">.</font><b><font color="#000000">extname</font></b><font color="#990000">(</font>filename<font color="#990000">);</font>
<b><font color="#0000FF">var</font></b> base <font color="#990000">=</font> filename<font color="#990000">.</font><b><font color="#000000">replace</font></b><font color="#990000">(</font>ext<font color="#990000">,</font> <font color="#FF0000">''</font><font color="#990000">);</font>
<b><font color="#0000FF">var</font></b> checksumFile <font color="#990000">=</font> base <font color="#990000">+</font> <font color="#FF0000">'-'</font> <font color="#990000">+</font> sum <font color="#990000">+</font> ext<font color="#990000">;</font>
<b><font color="#000000">callback</font></b><font color="#990000">(</font><b><font color="#0000FF">null</font></b><font color="#990000">,</font> <font color="#FF0000">{</font>
path<font color="#990000">:</font> file<font color="#990000">.</font>path<font color="#990000">,</font>
originalFile<font color="#990000">:</font> file<font color="#990000">.</font>originalFile<font color="#990000">,</font>
checksumFile<font color="#990000">:</font> checksumFile
<font color="#FF0000">}</font><font color="#990000">);</font>
<font color="#FF0000">}</font><font color="#990000">);</font>
<font color="#FF0000">}</font>
</tt></pre></blockquote>
<h2>Upload files to S3</h2>
<p>When the new filenames have been created the files can now be uploaded to S3
via <code>s3.putObject</code>. Unfortunately, <code>putObject</code> does not support <code>pipe</code>, but I
can use a ReadStream as the value of the body object and this is good enough.
It uses the <code>mime</code> module to calculate the content-type from the filename.
After the file is uploaded an object with a mapping between the original name
and the URL is returned.</p>
<blockquote><pre><tt><b><font color="#0000FF">function</font></b> <b><font color="#000000">uploadFiles</font></b><font color="#990000">(</font>prefix<font color="#990000">,</font> files<font color="#990000">,</font> callback<font color="#990000">)</font> <font color="#FF0000">{</font>
console<font color="#990000">.</font><b><font color="#000000">log</font></b><font color="#990000">(</font><font color="#FF0000">'uploadFiles'</font><font color="#990000">,</font> prefix<font color="#990000">,</font> files<font color="#990000">)</font>
async<font color="#990000">.</font><b><font color="#000000">map</font></b><font color="#990000">(</font>files<font color="#990000">,</font> uploadFile<font color="#990000">.</font><b><font color="#000000">bind</font></b><font color="#990000">(</font><b><font color="#0000FF">null</font></b><font color="#990000">,</font> prefix<font color="#990000">),</font> callback<font color="#990000">);</font>
<font color="#FF0000">}</font>
<b><font color="#0000FF">function</font></b> <b><font color="#000000">uploadFile</font></b><font color="#990000">(</font>prefix<font color="#990000">,</font> file<font color="#990000">,</font> callback<font color="#990000">)</font> <font color="#FF0000">{</font>
<b><font color="#0000FF">var</font></b> stream <font color="#990000">=</font> fs<font color="#990000">.</font><b><font color="#000000">createReadStream</font></b><font color="#990000">(</font>file<font color="#990000">.</font>path<font color="#990000">);</font>
<b><font color="#0000FF">var</font></b> s3options <font color="#990000">=</font> <font color="#FF0000">{</font>
Bucket<font color="#990000">:</font> config<font color="#990000">.</font>bucket<font color="#990000">,</font>
Key<font color="#990000">:</font> prefix <font color="#990000">+</font> file<font color="#990000">.</font>checksumFile<font color="#990000">,</font>
Body<font color="#990000">:</font> stream<font color="#990000">,</font>
ContentType<font color="#990000">:</font> mime<font color="#990000">.</font><b><font color="#000000">lookup</font></b><font color="#990000">(</font>file<font color="#990000">.</font>path<font color="#990000">)</font>
<font color="#FF0000">}</font><font color="#990000">;</font>
s3<font color="#990000">.</font><b><font color="#000000">putObject</font></b><font color="#990000">(</font>s3options<font color="#990000">,</font> <b><font color="#0000FF">function</font></b><font color="#990000">(</font>err<font color="#990000">,</font> data<font color="#990000">)</font> <font color="#FF0000">{</font>
<b><font color="#0000FF">if</font></b> <font color="#990000">(</font>err<font color="#990000">)</font> <b><font color="#0000FF">return</font></b> <b><font color="#000000">callback</font></b><font color="#990000">(</font>err<font color="#990000">);</font>
console<font color="#990000">.</font><b><font color="#000000">log</font></b><font color="#990000">(</font><font color="#FF0000">'Object added'</font><font color="#990000">,</font> s3options<font color="#990000">);</font>
<b><font color="#000000">callback</font></b><font color="#990000">(</font><b><font color="#0000FF">null</font></b><font color="#990000">,</font> <font color="#FF0000">{</font>
originalFile<font color="#990000">:</font> file<font color="#990000">.</font>originalFile<font color="#990000">,</font>
url<font color="#990000">:</font> config<font color="#990000">.</font>url <font color="#990000">+</font> config<font color="#990000">.</font>bucket <font color="#990000">+</font> <font color="#FF0000">'/'</font> <font color="#990000">+</font> prefix <font color="#990000">+</font> file<font color="#990000">.</font>checksumFile
<font color="#FF0000">}</font><font color="#990000">);</font>
<font color="#FF0000">}</font><font color="#990000">);</font>
<font color="#FF0000">}</font>
</tt></pre></blockquote>
<h2>Upload the index</h2>
<p>The last thing to is to upload the index with the filename-to-URL map as a
JSON-file. This is done in a similar way as the upload of the images.</p>
<blockquote><pre><tt><b><font color="#0000FF">function</font></b> <b><font color="#000000">uploadIndex</font></b><font color="#990000">(</font>prefix<font color="#990000">,</font> files<font color="#990000">,</font> callback<font color="#990000">)</font> <font color="#FF0000">{</font>
<b><font color="#0000FF">var</font></b> s3options <font color="#990000">=</font> <font color="#FF0000">{</font>
Bucket<font color="#990000">:</font> config<font color="#990000">.</font>bucket<font color="#990000">,</font>
Key<font color="#990000">:</font> prefix <font color="#990000">+</font> <font color="#FF0000">'/index.json'</font><font color="#990000">,</font>
Body<font color="#990000">:</font> JSON<font color="#990000">.</font><b><font color="#000000">stringify</font></b><font color="#990000">(</font>files<font color="#990000">),</font>
ContentType<font color="#990000">:</font> <font color="#FF0000">'application/json'</font>
<font color="#FF0000">}</font><font color="#990000">;</font>
s3<font color="#990000">.</font><b><font color="#000000">putObject</font></b><font color="#990000">(</font>s3options<font color="#990000">,</font> <b><font color="#0000FF">function</font></b><font color="#990000">(</font>err<font color="#990000">,</font> data<font color="#990000">)</font> <font color="#FF0000">{</font>
<b><font color="#0000FF">if</font></b> <font color="#990000">(</font>err<font color="#990000">)</font> <b><font color="#0000FF">return</font></b> <b><font color="#000000">callback</font></b><font color="#990000">(</font>err<font color="#990000">);</font>
console<font color="#990000">.</font><b><font color="#000000">log</font></b><font color="#990000">(</font><font color="#FF0000">'Object added'</font><font color="#990000">,</font> s3options<font color="#990000">.</font>Key<font color="#990000">);</font>
<b><font color="#000000">callback</font></b><font color="#990000">(</font><b><font color="#0000FF">null</font></b><font color="#990000">,</font> <font color="#FF0000">{</font>
files<font color="#990000">:</font> files<font color="#990000">,</font>
url<font color="#990000">:</font> config<font color="#990000">.</font>url <font color="#990000">+</font> config<font color="#990000">.</font>bucket <font color="#990000">+</font> <font color="#FF0000">'/'</font> <font color="#990000">+</font> prefix <font color="#990000">+</font> <font color="#FF0000">'/index.json'</font>
<font color="#FF0000">}</font><font color="#990000">);</font>
<font color="#FF0000">}</font><font color="#990000">);</font>
<font color="#FF0000">}</font>
</tt></pre></blockquote>
<p>The final index.json file loooks something like this.</p>
<blockquote><pre><tt><font color="#990000">[</font><font color="#FF0000">{</font>
originalFile<font color="#990000">:</font> <font color="#FF0000">"/Tapir_standing_profile.jpg"</font><font color="#990000">,</font>
url<font color="#990000">:</font> <font color="#FF0000">"https://s3-eu-west-1.amazonaws.com/anders-dest/tapirs/Tapir_standing_profile-624bd0ac55d5140a78a2ea9d1409e2f6.jpg"</font>
<font color="#FF0000">}</font><font color="#990000">,</font>
<font color="#FF0000">{</font>
originalFile<font color="#990000">:</font> <font color="#FF0000">"/tapir-sticker.png"</font><font color="#990000">,</font>
url<font color="#990000">:</font> <font color="#FF0000">"https://s3-eu-west-1.amazonaws.com/anders-dest/tapirs/tapir-sticker-8522f4228bbc995d73ee1ead9d5e8e4f.png"</font>
<font color="#FF0000">}</font><font color="#990000">,</font>
<font color="#FF0000">{</font>
originalFile<font color="#990000">:</font> <font color="#FF0000">"/tapir.jpg"</font><font color="#990000">,</font>
url<font color="#990000">:</font> <font color="#FF0000">"https://s3-eu-west-1.amazonaws.com/anders-dest/tapirs/tapir-eb09705a33f6c6896def4e452fa77272.jpg"</font>
<font color="#FF0000">}</font><font color="#990000">]</font>
</tt></pre></blockquote>
<h2>Summary</h2>
<p>Lambda is very simple to work with and it allows me to create small services
that react to events without the need to setup any servers at all.</p>
<p>Apart from the integration with S3, it also integrates with Kinesis and with
DynamoDB allowing for very cool application to built.</p>
Anders Janmyrhttp://www.blogger.com/profile/16081537018884349622noreply@blogger.com5tag:blogger.com,1999:blog-34049130.post-37638030646589494432014-09-27T12:00:00.000+02:002014-09-27T12:00:06.532+02:00Fallacies and Biases of our Imperfect Mind
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgDiDmCKuM02VHVi-u5e-fKfBcQwg_Btjl_y3_s5oSPM7RpObk14YJedRoyJJzMjP6-BVak-frGMm3foOECEBpjSooa6aK0xdAlMcT0MK6vGRUISWi4ZtrshMquLnRzZGDvRBtK/s1600/fallacies-mind.001.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgDiDmCKuM02VHVi-u5e-fKfBcQwg_Btjl_y3_s5oSPM7RpObk14YJedRoyJJzMjP6-BVak-frGMm3foOECEBpjSooa6aK0xdAlMcT0MK6vGRUISWi4ZtrshMquLnRzZGDvRBtK/s200/fallacies-mind.001.png" /></a></div>
<p>Our mind is the most advanced computer we know about. It can perform tremendous
feats. Yet, it is fooling us a lot more than most of us would care to admit.
The reason for this is that the mind takes shortcuts to save energy and speed up
our thinking.</p>
<p>In this article I will present how science now believes that the brain
works, the problems it has, and suggestions about what to do about it.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgfbvj1JcRYBQ42gEWtqD3E6WS_aVtvoPVaBzOmEZWzxVPhLsklBk2ZauxZwSUlXFA11w_WH18v6WEeBsUSGcEbJEucYL_UXWj3fv2IIdqsH-_N-MgFvBo1WDirO5-pCFRhXWCQ/s1600/fallacies-mind.002.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgfbvj1JcRYBQ42gEWtqD3E6WS_aVtvoPVaBzOmEZWzxVPhLsklBk2ZauxZwSUlXFA11w_WH18v6WEeBsUSGcEbJEucYL_UXWj3fv2IIdqsH-_N-MgFvBo1WDirO5-pCFRhXWCQ/s200/fallacies-mind.002.png" /></a></div>
<h2>Our Incredible Mind</h2>
<p>Imagine you are riding a bicycle into an intersection. Cars, motorcycles,
mopeds and other bikes are coming from all directions. Your brain takes in the
whole scene and makes instantaneous decisions about what route to take. You
communicate both consciously and unconsciously with the other drivers and you
cross the intersection as if it was nothing.</p>
<p>This is an example of what our incredible mind can do. But, in order to do this
it takes shortcuts and these shortcuts are not always appropriate. The rest of
this article will discuss the problems that occur when the shortcuts are not to
our advantage.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEinSmiukEyxDtJvtcV2xDGc-RK3lxPAy5uq5u4VfSxZe268KPTbTMGCOngLplKz-AeboO2m4mfN2VjHhJDHu5CxCjKkkE_F8ST7X6Cz3CtfFY1Qe4Rf8R3Q0IkTDbXXXoRXQAxy/s1600/fallacies-mind.003.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEinSmiukEyxDtJvtcV2xDGc-RK3lxPAy5uq5u4VfSxZe268KPTbTMGCOngLplKz-AeboO2m4mfN2VjHhJDHu5CxCjKkkE_F8ST7X6Cz3CtfFY1Qe4Rf8R3Q0IkTDbXXXoRXQAxy/s200/fallacies-mind.003.png" /></a></div>
<h2>Belief</h2>
<p>What is belief? Why do we believe the things we do? What do we truly know? When
we start to really analyze our beliefs we often realize that we don't know why
we believe in something, we just do. And, we may also know that something is
not correct but still act as if it is.</p>
<p>Can you get a cold from being cold?</p>
<p>No! The only way to get a cold is by being exposed to the cold virus. If you
catch a cold after being cold it is only a coincidence. Yet, many of us tell
our children to dress warmly to avoid getting a cold!</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhys_RgySugEdegNFsGiRM2qyzPv6lhXhzPSjerI4VGW8j1QzLLY79VMtvnRxEKv36SsvhIzSWKWc76L4AP0fRqBdGoh61_UCE-Yuilj9oUK3jRrqmMftAyQ043H4eRo8HvVs5j/s1600/fallacies-mind.004.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhys_RgySugEdegNFsGiRM2qyzPv6lhXhzPSjerI4VGW8j1QzLLY79VMtvnRxEKv36SsvhIzSWKWc76L4AP0fRqBdGoh61_UCE-Yuilj9oUK3jRrqmMftAyQ043H4eRo8HvVs5j/s200/fallacies-mind.004.png" /></a></div>
<h2>Perception</h2>
<p>We think that our perception is infallible. We think we see what is real! This
is not the case, our senses are easily fooled and also affected by what we
expect to experience.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi-b7MJTKUfdEuihY3Aa54xjLYTkZ5RqADu-UheI-FUBC67-YAjL9KpGVZQ8ojAm7lMDS-N0m3Kh8wHajnH0f5ssSGNtF4JN5OJdLqHBeqWmTo0owps6IqERmUHIl8UCYqByzH4/s1600/fallacies-mind.005.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi-b7MJTKUfdEuihY3Aa54xjLYTkZ5RqADu-UheI-FUBC67-YAjL9KpGVZQ8ojAm7lMDS-N0m3Kh8wHajnH0f5ssSGNtF4JN5OJdLqHBeqWmTo0owps6IqERmUHIl8UCYqByzH4/s200/fallacies-mind.005.png" /></a></div>
<h2>Shadow Illusion</h2>
<p>Which one of A and B is lighter?</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiOsJfkzROPN7CTyzEfEbrcuz-Kn9FkhNiX5m-g_W26NyGi2vo11_1oMUouOI7CFBtnflBrDK7AhcNna_6n2V4qf1_ZHDpM7cdhn-RjiBEQHsQjNFwPlcE1l7114hg4NLrg7NsW/s1600/fallacies-mind.006.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiOsJfkzROPN7CTyzEfEbrcuz-Kn9FkhNiX5m-g_W26NyGi2vo11_1oMUouOI7CFBtnflBrDK7AhcNna_6n2V4qf1_ZHDpM7cdhn-RjiBEQHsQjNFwPlcE1l7114hg4NLrg7NsW/s200/fallacies-mind.006.png" /></a></div>
<p>It is a trick question, they are both the same color as we see in this picture.
Yet, even when we know this, it is impossible to see!</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhFpCu63MuH0ViEPs59FVut7J44jsObsZI1raCHYbRR8wSUIKJb_xrwiKKzYSqw6iA2uLphbvCy321YkOXhuuJ-d1yBtRzO8AirqB46TUm_Hx0ZXLqpYEinYMcfO_8RXXqtscx9/s1600/fallacies-mind.007.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhFpCu63MuH0ViEPs59FVut7J44jsObsZI1raCHYbRR8wSUIKJb_xrwiKKzYSqw6iA2uLphbvCy321YkOXhuuJ-d1yBtRzO8AirqB46TUm_Hx0ZXLqpYEinYMcfO_8RXXqtscx9/s200/fallacies-mind.007.png" /></a></div>
<h2>Pattern Illusion</h2>
<p>Can you see anything in this image? Can you see the dalmatian?</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhm9sbIEMj1zeb72SaywttEDR7YT_GofNBzt_gmODtL0kZsrPbOwLeF_AGRutA7MZQrxHbv1pByN3NzOkWYpQzhMpSk7YcwNH9SmBiOw5lKXineKtN6mCCSAgTt1_N8u_14EUBe/s1600/fallacies-mind.008.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhm9sbIEMj1zeb72SaywttEDR7YT_GofNBzt_gmODtL0kZsrPbOwLeF_AGRutA7MZQrxHbv1pByN3NzOkWYpQzhMpSk7YcwNH9SmBiOw5lKXineKtN6mCCSAgTt1_N8u_14EUBe/s200/fallacies-mind.008.png" /></a></div>
<p>If we draw the contour of the dalmatian it becomes obvious. But now, if you
look at the above picture. Can you <strong>not</strong> see the dalmatian? Our perception is
influenced by what we expect to see.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjj2d8sAoByuTITxaQ2FCEq61sU1LYGEBHkSHhwrpdpA6S5TPbp1IjQ5mxpcbYYdEBMqsu8WJgI9wKDGoUdBakgOa1qFmXYgU2W_ifjj7SNnhBZomLa2tvc4dzIZ-yEko1Yrq7J/s1600/fallacies-mind.009.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjj2d8sAoByuTITxaQ2FCEq61sU1LYGEBHkSHhwrpdpA6S5TPbp1IjQ5mxpcbYYdEBMqsu8WJgI9wKDGoUdBakgOa1qFmXYgU2W_ifjj7SNnhBZomLa2tvc4dzIZ-yEko1Yrq7J/s200/fallacies-mind.009.png" /></a></div>
<h2>Attention Test</h2>
<p>Watch <a href='https://www.youtube.com/watch?v=vJG698U2Mvo'>this film</a> and try to count the passes made by the white-dressed basketball
players.</p>
<p>Did you get the count right? Did you see the gorilla? In the original study
about half of the people that were shown this film didn't see the gorilla!
Being focused on one thing can make us completely miss another.</p>
<p>This happens to us all the time in real life. People look at the same situation
and interpret it completely differently.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiyL83q4aeZI9wU410A2Y4fJQLmS2mVOdzYJnSqiZE9IybfuEs_xGDG-Et2RVh-oRe7l9sl7zBIUiOBzcvfY1OypzEJiL3sQPQRNvE2SKDt5ddK-7dAlvPO2pwW04n_Lkb2dlZG/s1600/fallacies-mind.010.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiyL83q4aeZI9wU410A2Y4fJQLmS2mVOdzYJnSqiZE9IybfuEs_xGDG-Et2RVh-oRe7l9sl7zBIUiOBzcvfY1OypzEJiL3sQPQRNvE2SKDt5ddK-7dAlvPO2pwW04n_Lkb2dlZG/s200/fallacies-mind.010.png" /></a></div>
<h2>Memory</h2>
<p>We believe that we remember things as they actually were, but in reality our
memories are reconstructed every time we remember something. We fill in new
details.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh3Q39YeUOJ6vDRhCH4dEeV9fzh80-zOSOg8a69eNGWCqF_yQT4yBjV6NqeAjGXUQXQsQ0jyqi9IQoazDmqC_E6cX1NqYrv8EJb8FKlKws3msEvz_BYlsitLQurVthbieX4gPjm/s1600/fallacies-mind.011.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh3Q39YeUOJ6vDRhCH4dEeV9fzh80-zOSOg8a69eNGWCqF_yQT4yBjV6NqeAjGXUQXQsQ0jyqi9IQoazDmqC_E6cX1NqYrv8EJb8FKlKws3msEvz_BYlsitLQurVthbieX4gPjm/s200/fallacies-mind.011.png" /></a></div>
<h2>Source and Truth Amnesia</h2>
<p>We have a tendency to forget the source and the truthiness about facts that we
<em>know</em>. We remember the facts, but we don't know where they come from or if
they are true or not!</p>
<p>We may have heard about a correlation between vaccines and autism. But, we
forgot, the minor detail, that there is a not even a very weak correlation
between them. Hence, we refuse to vaccinate our kids since we don't want them
to become autistic.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhI92OHbzrxSxfxGq8-oOOiRhGOfvaJp6uT9vU7L6RvWiaw7EzADmJh4vgNDNnheggvFFjrVQJzhJvr6v4iVXph2hvO8b_CO0cKC98ReUBDfut1ZWTY2mz8zIvX-XwEysmWphbh/s1600/fallacies-mind.012.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhI92OHbzrxSxfxGq8-oOOiRhGOfvaJp6uT9vU7L6RvWiaw7EzADmJh4vgNDNnheggvFFjrVQJzhJvr6v4iVXph2hvO8b_CO0cKC98ReUBDfut1ZWTY2mz8zIvX-XwEysmWphbh/s200/fallacies-mind.012.png" /></a></div>
<h2>Vivid Memories</h2>
<p>Vivid memories, memories involving strong feelings, makes us remember things
more strongly. It makes us more confident about our memories being correct.
Just because the memories are stronger does not mean that they are more
correct. We simply believe in them more.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhAnBPtwmTX-I6zBhv_1hXkvSvgxtWwPlx0BUT6_oNvHng9a9CJI5cBHSQ-avN3hyphenhyphenve7GHXBl0k5e2k1H1JhnvaSE6KjyNG9U6_Cjpd2irMwWRmXslj7k9PfHmDcneDbbvza8Kt/s1600/fallacies-mind.013.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhAnBPtwmTX-I6zBhv_1hXkvSvgxtWwPlx0BUT6_oNvHng9a9CJI5cBHSQ-avN3hyphenhyphenve7GHXBl0k5e2k1H1JhnvaSE6KjyNG9U6_Cjpd2irMwWRmXslj7k9PfHmDcneDbbvza8Kt/s200/fallacies-mind.013.png" /></a></div>
<h2>Memory Fusion</h2>
<p>Memories also fuse together to form new composite memories, that may not
resemble what really happened at all. Do you remember your tenth birthday or
do you remember what your mom told you or what you have seen in pictures? </p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgBqRzF5zdMeRFyfBIkw9HaYs9qDGJ7qpVmNYDJQ6w51ZDOjBs3iLLdGbS6wr5o815lBeyUoL-W7J5GTzSwS8ycV7QTSix2_ADgYF7c7lyvTxXq8mj9Veerf0GL0jA0_Ie8rTUe/s1600/fallacies-mind.014.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgBqRzF5zdMeRFyfBIkw9HaYs9qDGJ7qpVmNYDJQ6w51ZDOjBs3iLLdGbS6wr5o815lBeyUoL-W7J5GTzSwS8ycV7QTSix2_ADgYF7c7lyvTxXq8mj9Veerf0GL0jA0_Ie8rTUe/s200/fallacies-mind.014.png" /></a></div>
<h2>Fake Memories</h2>
<p>We cannot tell if our memories are fake or if they really happened. Everything
we remember seems real to us!</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQt_pY6f8d-NKG-OEwCRyL1sVOe7uP6uUA0_DOIGh7QffSDz8jWVUGsyPnFBGUO_J5UjylQXlK9oaE9TKpuyYiA0YbXzV6wxumqMGqVv9uuAV35pb8REgLvZIMESTKZmP_VPoP/s1600/fallacies-mind.015.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQt_pY6f8d-NKG-OEwCRyL1sVOe7uP6uUA0_DOIGh7QffSDz8jWVUGsyPnFBGUO_J5UjylQXlK9oaE9TKpuyYiA0YbXzV6wxumqMGqVv9uuAV35pb8REgLvZIMESTKZmP_VPoP/s200/fallacies-mind.015.png" /></a></div>
<h2>Pattern Recognition</h2>
<p>Humans are also very good at pattern recognition. This allow us to detect and
categorize people, animals, and things. But, our pattern recognition also shows
us things that are not there. Was there really a dalmatian in the spotted
picture above?</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9eGxqflNpOwxFc2RMK05SKz5KFS1lpPZmD0cCJrAZrnitj7rOPn9JhsdwqmXcww2-W8WkMDm0sHfdHWosUQFMRu8vMf8xD8w-swj-AZ3RyRbX9TusY0gwjI4x42OmmoQLppDc/s1600/fallacies-mind.016.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9eGxqflNpOwxFc2RMK05SKz5KFS1lpPZmD0cCJrAZrnitj7rOPn9JhsdwqmXcww2-W8WkMDm0sHfdHWosUQFMRu8vMf8xD8w-swj-AZ3RyRbX9TusY0gwjI4x42OmmoQLppDc/s200/fallacies-mind.016.png" /></a></div>
<h2>Agent Detection</h2>
<p>Agent detection is an inclination for humans and animals to detect an
intelligent agent in situations that may or may not involve one. We see and
hear things that aren't there.</p>
<p>We detect a bush blowing in the wind as a person hiding. We see a rope lying on
the trail as a dangerous snake.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbXWINJLj_Uwm0oUZubioXA1e609LBI7yHTJtgueXtLEnTgqATOrafbczwcCSV3DordiPk6MWOFIjjtDmczXkcoXu4_jsOFSZP1G6l43Yv-w5KHdYo2KwP_vcwbVO_hHXS6ZPA/s1600/fallacies-mind.017.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbXWINJLj_Uwm0oUZubioXA1e609LBI7yHTJtgueXtLEnTgqATOrafbczwcCSV3DordiPk6MWOFIjjtDmczXkcoXu4_jsOFSZP1G6l43Yv-w5KHdYo2KwP_vcwbVO_hHXS6ZPA/s200/fallacies-mind.017.png" /></a></div>
<h2>Confabulated Consciousness</h2>
<p>Our mind processes our perceptions and memories and creates our reality into a
coherent story. The story need not be correct, it must only be consistent. In
order to keep the story consistent our mind makes up the details it needs to.</p>
<p>In a study of split-brain patients, the patients were shown images. One image
per eye. The split-brain condition prevents the two parts of the brain from
communicating with each other.</p>
<p>In the depicted example, the patient was shown two images: one eye was shown a
chicken foot, the other eye was shown a snowy landscape. The patient then had
to pick a related image from a number of other pictures. The patient in the study
picked a hen and a snow-shovel with each hand respectively.</p>
<p>When asked why he picked the images, his verbal side of the brain answered. "I
picked the hen because I saw a chicken's foot and I picked the shovel because I
need a shovel to clean out the hen house."</p>
<p>His mind made up story that was consistent with why he had a shovel in his
other hand.</p>
<p><em>Our mind can make things up to make our life story consistent!</em></p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhS7ATgx3orr0EEP0WsCoEnXkOpHG81SkhrRh88gp20Wo3XymZNHhEqQZyNDgr4nY3oXgX9WRserPH9D3jedY8Xo717QVNH81g2_OQIZ776hJnBTy7Kz62cYmPk3lpLt50O0sFR/s1600/fallacies-mind.018.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhS7ATgx3orr0EEP0WsCoEnXkOpHG81SkhrRh88gp20Wo3XymZNHhEqQZyNDgr4nY3oXgX9WRserPH9D3jedY8Xo717QVNH81g2_OQIZ776hJnBTy7Kz62cYmPk3lpLt50O0sFR/s200/fallacies-mind.018.png" /></a></div>
<h2>Bias</h2>
<p>A bias is a prejudice. A cognitive bias is a type of error in thinking that
occurs when we are processing and interpreting information in the world around
us.</p>
<p>Cognitive biases are often a result of our attempt to simplify information
processing. They are rules of thumb that help us make sense of the world and
reach decisions with relative speed.</p>
<p>Unfortunately, these biases sometimes trip us up, leading to poor decisions and
bad judgments.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirROwsjx3YURnW403TVWLt2SMWacqT3xcPa13ZNwcUil2Hjz3MOCwd-M5PIdqpYMuvb5R_AWRqwNpww6LSnu_AgxGCWt3FQZRUD0w8vlpCICazOw4p5dMuqRvozJkVMLIwHrku/s1600/fallacies-mind.019.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirROwsjx3YURnW403TVWLt2SMWacqT3xcPa13ZNwcUil2Hjz3MOCwd-M5PIdqpYMuvb5R_AWRqwNpww6LSnu_AgxGCWt3FQZRUD0w8vlpCICazOw4p5dMuqRvozJkVMLIwHrku/s200/fallacies-mind.019.png" /></a></div>
<h2>Anchoring</h2>
<p>The anchoring effect describes the human tendency to rely to heavily on the
first piece of information offered, the anchor, when making decisions.</p>
<p>If I ask a group of people "If more or less than 20 percent of the mammals
have four legs?" and then ask the same group to guess the specific percentage
of animals that have four legs. I commonly get a lower percentage than if I
initially had asked "If more of less than 80 percent of the mammals have four
legs?".</p>
<p>We anchor to the number presented to us. This is the same technique that is
used by salesmen when they offer you a good deal of only 20 thousand dollars
for the second-hand Volvo.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJ_mX_U2931dCuzadpR7TfJZg14jACe6RQoiE3HIlcoxoircGhIFvNEq3YVU1PoPe76RvVaggwWKGWOJA5b6hX_ru7Z1F2QZh6c6b9SZb0l0Ws00UioC06jZU8k3WnXLeUqPQo/s1600/fallacies-mind.020.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJ_mX_U2931dCuzadpR7TfJZg14jACe6RQoiE3HIlcoxoircGhIFvNEq3YVU1PoPe76RvVaggwWKGWOJA5b6hX_ru7Z1F2QZh6c6b9SZb0l0Ws00UioC06jZU8k3WnXLeUqPQo/s200/fallacies-mind.020.png" /></a></div>
<h2>Availability Heuristic</h2>
<p>How many percent of the population do you think are allergic to gluten? How do
you go about making such an estimation? What I often do is to think about the
people around me. How many of them are allergic to gluten? It seems like quite
a lot. I would guess about 10 percent of the people I know are allergic, so
that is my reply.</p>
<p>This is the availability heuristic at work. Why should my tiny number of
acquaintances have anything to do with the rest of the population in the world?
But, this information is readily available to me and it is easier for me to
just guess from this information than to think through the problem thoroughly.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5V0QCK5xfoy43ww73b5ed1vLc3fv5cDukZPuVEFFqQrvFaF1kc-CZqum8FJPNyAz3BbVxkks-2fplT5lblXM8v2AV2uHug5vb4NeuCQXQzb8w1iV67ENIOfs0hDoirHOPFQxj/s1600/fallacies-mind.021.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5V0QCK5xfoy43ww73b5ed1vLc3fv5cDukZPuVEFFqQrvFaF1kc-CZqum8FJPNyAz3BbVxkks-2fplT5lblXM8v2AV2uHug5vb4NeuCQXQzb8w1iV67ENIOfs0hDoirHOPFQxj/s200/fallacies-mind.021.png" /></a></div>
<h2>Fundamental Attribution Error</h2>
<p>Say you are walking in the street and stumble and fall. The common way we react
to this is that we make up an excuse to why we fell, a hole in the pavement, etc.
It is not my fault, there was a hole in the pavement. Perhaps, we even get
angry, someone should really fix that!</p>
<p>If someone else stumbles and falls in the same spot, we readily label that
person as being clumsy or careless.</p>
<p>We attribute our mistakes to external causes and other's mistakes to the
person. We also give ourselves credit for good things we do, but other people's
good deeds we attribute to luck or coincidence. This is the fundamental
attribution error.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjjwPpfzJy2zx3jXVrSxudPMAT431vBkDw_huaBMwuokL-TvsSXzcp4JSdHp0-VbYbHaegUhD1IU-R1YO87Uj8DdMIpkdT_cM7AKXJo4FGOALZCg3LPvvzGJWRdFm9zZEuOlCiu/s1600/fallacies-mind.022.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjjwPpfzJy2zx3jXVrSxudPMAT431vBkDw_huaBMwuokL-TvsSXzcp4JSdHp0-VbYbHaegUhD1IU-R1YO87Uj8DdMIpkdT_cM7AKXJo4FGOALZCg3LPvvzGJWRdFm9zZEuOlCiu/s200/fallacies-mind.022.png" /></a></div>
<h2>Hindsight Bias</h2>
<p>Hindsight bias is also known as the "I-knew-it-all-along" effect. It is the
tendency to see past events as being predictable at the time those events
happened. (This picture does not really convey this bias, as the outcome can
probably be predicted beforehand :)</p>
<p>An example of this is the 9/11 bombings, when the event had happened it was
easy to find clues that informed about a coming attack. Clues like this exist
all the time for things that never happen, but we don't focus on those because
they are not relevant.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwfGKYyxW6PF49i-oFwdq3wKzaHonSApeR53p6VqdeVz-ndEn7-u0iDKVa8A2YItbJxUYe7yT_fN0QEVfhkXVNjsrlVuSWfzNtTwBoEk_AUC8QaUYjq-Q2CuFUP4n4Zep4tWBL/s1600/fallacies-mind.023.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwfGKYyxW6PF49i-oFwdq3wKzaHonSApeR53p6VqdeVz-ndEn7-u0iDKVa8A2YItbJxUYe7yT_fN0QEVfhkXVNjsrlVuSWfzNtTwBoEk_AUC8QaUYjq-Q2CuFUP4n4Zep4tWBL/s200/fallacies-mind.023.png" /></a></div>
<h2>Confirmation Bias</h2>
<p>This is the mother of all biases! A bias that we, all of us, fall into every
day. It is the tendency to search for or interpret information in a way that
confirms our beliefs. Or, to notice events that confirms our beliefs while
ignoring events that disconfirms them.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiX6mZSUZKUGkfMV1yv_5wzuXqu3v6YGiOroYk4XY4PeyZMWFnwlUv6L5lknl3G2IBvV_i-JyGlTGPhyzaJMtgXtt402nQcnAscTM0ys__VPE8aA9-wqTR-oXkwgwkGMb7jDEgd/s1600/fallacies-mind.024.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiX6mZSUZKUGkfMV1yv_5wzuXqu3v6YGiOroYk4XY4PeyZMWFnwlUv6L5lknl3G2IBvV_i-JyGlTGPhyzaJMtgXtt402nQcnAscTM0ys__VPE8aA9-wqTR-oXkwgwkGMb7jDEgd/s200/fallacies-mind.024.png" /></a></div>
<p>Do I put the seat down when I have been on the toilet? All the time, I say.
Never, my wife says. How can this be? How can I and my wife come to completely
different conclusions from the same data?</p>
<p>The reason is that I notice the times when I remember to put the seat down,
since I have to think about this and therefore remember it. I don't remember
the times when I don't do it since, I don't even notice them.</p>
<p>For my wife it is the absolute opposite, she only notices when I forget to do
it and doesn't notice when I do.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXIMHBLA3B93rWVjNMwehP_d-6LG73ATE4TcdSGatptyNWS2OURxCELwWx8DihNqIcmdnKookVziud_AoEhbx1I18swV00_rUQsKWu0wdh9MP46qcAspKB0xkk1BLvD4bRN55a/s1600/fallacies-mind.025.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXIMHBLA3B93rWVjNMwehP_d-6LG73ATE4TcdSGatptyNWS2OURxCELwWx8DihNqIcmdnKookVziud_AoEhbx1I18swV00_rUQsKWu0wdh9MP46qcAspKB0xkk1BLvD4bRN55a/s200/fallacies-mind.025.png" /></a></div>
<p>When we read an article that we agree with, it is easy to think, "Yes, that is
the way it is!" and move on. If we read an article that we don't agree with, we
can go to great lengths to examine the "erroneous" arguments to disconfirm
them.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2-GldhaC5H3akpxAc3rJp26_tzCTtE2g59DFGFw7S1inhnqUSQMQesWaGB8Zj7raMDsa6aCv36uXaU2DSg_ZRM_FAVkFg0nAI9PaYveJTcFpVgeTuC0_ZctsCaM3z-dbVsmN7/s1600/fallacies-mind.026.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2-GldhaC5H3akpxAc3rJp26_tzCTtE2g59DFGFw7S1inhnqUSQMQesWaGB8Zj7raMDsa6aCv36uXaU2DSg_ZRM_FAVkFg0nAI9PaYveJTcFpVgeTuC0_ZctsCaM3z-dbVsmN7/s200/fallacies-mind.026.png" /></a></div>
<h2>Innumeracy</h2>
<p>The human mind is really bad at working with large numbers and probability.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgemASBa-tr5XZW85FgMexFIEfK2BGkjdwb251K1Myv5CHISqY6zlPdMPhau0G4IXU2wCtf0DfaObUtr8kAyxrQ9dJNMdyduv-dVvI840dj2vcQQhWbkJjBssVR0F95Bze1QVRT/s1600/fallacies-mind.027.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgemASBa-tr5XZW85FgMexFIEfK2BGkjdwb251K1Myv5CHISqY6zlPdMPhau0G4IXU2wCtf0DfaObUtr8kAyxrQ9dJNMdyduv-dVvI840dj2vcQQhWbkJjBssVR0F95Bze1QVRT/s200/fallacies-mind.027.png" /></a></div>
<h2>Gambler's Fallacy</h2>
<p>The tendency to think that future probabilities are altered by past events,
when in reality they are unchanged.</p>
<p>Flip a coin ten times in a row and it turns out tails every time? How likely
is it that we will flop a heads the next time. The answer is, of course, 50%.
In this scenario most of us will know this is correct, but in many other
scenarios we tend to think that the other option is due and hence calculate it
as more likely to occur.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgire1RVLqW_wAanvClMFll3Cfv2t72xjOvzm1MbQ0Bc77uCoVyng2gBidB-quK6o2ws4fbUrNCekcu8AbGMqU_ACgIRCHq-lBRPpCOR7kY-XLPX3oZz1RHjR150_hd8BnKJ3ol/s1600/fallacies-mind.028.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgire1RVLqW_wAanvClMFll3Cfv2t72xjOvzm1MbQ0Bc77uCoVyng2gBidB-quK6o2ws4fbUrNCekcu8AbGMqU_ACgIRCHq-lBRPpCOR7kY-XLPX3oZz1RHjR150_hd8BnKJ3ol/s200/fallacies-mind.028.png" /></a></div>
<h2>Lottery Fallacy</h2>
<p>What is the odds of <em>one person</em> winning a lottery? Not very high, maybe
one-in-a-billion, depending on which lottery it is. But often times this is not
the real question to ask ourselves. We should often ask: What is the odds of
<em>anyone</em> winning the lottery? It turns out that the odds for this are, usually,
pretty good.</p>
<p>Imagine you dream that someone dies. When you wake up the next day it has
really happened. What are the odds of this happening to you? I must be a
miracle. No! The correct question is: What are the adds of this happening to
anyone?</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXspykojtG9u1dQ9mcfBfklFXAGm0bwAnbQKiYTeubd9KB7ao8fpRRnbDRaqe8x16699wHHW1gKjLg86tSun-tyNqxniopGKxXl9r3bg6YpQQm8x0gXfYE3MwmnKmsM-Cq6uot/s1600/fallacies-mind.029.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXspykojtG9u1dQ9mcfBfklFXAGm0bwAnbQKiYTeubd9KB7ao8fpRRnbDRaqe8x16699wHHW1gKjLg86tSun-tyNqxniopGKxXl9r3bg6YpQQm8x0gXfYE3MwmnKmsM-Cq6uot/s200/fallacies-mind.029.png" /></a></div>
<h2>Base Rate Neglect</h2>
<p>John is a man who wears Gothic inspired clothing, has long black hair, and
listens to death metal. How likely is it that he is a Christian and how likely
is it that he is a Satanist?</p>
<p>We have a tendency to answer that it is more likely that he is a Satanist. But,
this ignores the base rate. The fact that there are 2 billion Christians and
only, maybe, 2 million Satanists. With that base rate in place, it is much more likely
that John is a Christian who likes wearing Gothic clothing, has long black
hair and listens to death metal.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg6xTWSN6SxHH4wkKR-bl6bOuO01euDDAo-0uwfsAjUkiMuvuU6slRXY3YNRUHtbDo9nVZ2kHPNth88hq8gQAhrgNXwmhsauMI9aaqEIDK3tcT3NhjCksZP_B1b8A1NRn5QyCUD/s1600/fallacies-mind.030.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg6xTWSN6SxHH4wkKR-bl6bOuO01euDDAo-0uwfsAjUkiMuvuU6slRXY3YNRUHtbDo9nVZ2kHPNth88hq8gQAhrgNXwmhsauMI9aaqEIDK3tcT3NhjCksZP_B1b8A1NRn5QyCUD/s200/fallacies-mind.030.png" /></a></div>
<h2>Clustering Illusion</h2>
<p>This is the tendency to overestimate the importance of small runs, streaks, or
clusters in large samples of random data.</p>
<p>The clustering illusion explains the "hot-hand" in basketball. The hot-hand is
the belief that a player who has made a few baskets is more likely to make the
next basket since he is on-a-roll.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi8RSiV5ty8284uN6ZAYfQzMXUFn7EQaTiB1EGXiHSNp-5f5EdvG9xjyw8XacwBzr47_f99p3FOdfnQ0H7w1_svoTqkgB_TiXsHLwkY6ozq-0NuXr87ohjIypid32m78acJiSg-/s1600/fallacies-mind.031.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi8RSiV5ty8284uN6ZAYfQzMXUFn7EQaTiB1EGXiHSNp-5f5EdvG9xjyw8XacwBzr47_f99p3FOdfnQ0H7w1_svoTqkgB_TiXsHLwkY6ozq-0NuXr87ohjIypid32m78acJiSg-/s200/fallacies-mind.031.png" /></a></div>
<h2>Probability</h2>
<p>Imagine a disease that 1% of the population has. Assume there is a test with
99% certainty of being correct. 1% false positives and 1% false negatives.</p>
<p>What is the probability that you have the disease if after taking the test it
shows positive?</p>
<p>Our natural inclination to answer this question is, "Bloody sure!". But, in
reality the probability of us having the disease is only 50%. Google it, if you
don't believe it!</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEge1c-a9fSkmFS6KxmPPPNnzgdjV6Yj_eilOWfZqyOZ5gK9mrcjH644Ni3gPkNKi4oTVQ8bAFDC7nE_jaCKthY1cVvBFQaPboqsa72vkVGiUQK8JUxLoX_6BvwJJmGK_sXW0QMm/s1600/fallacies-mind.032.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEge1c-a9fSkmFS6KxmPPPNnzgdjV6Yj_eilOWfZqyOZ5gK9mrcjH644Ni3gPkNKi4oTVQ8bAFDC7nE_jaCKthY1cVvBFQaPboqsa72vkVGiUQK8JUxLoX_6BvwJJmGK_sXW0QMm/s200/fallacies-mind.032.png" /></a></div>
<h2>So What?</h2>
<p>So, we believe things, but we don't know why. Our perception is severely
influenced by what we already believe. Our memories are flawed. We see patterns
and agents that don't exist. So what? This doesn't apply to us anyway, right?
</p>
<p>
It turns out that it does. Smarter people are better at rationalizing their beliefs than other's.
We still make the same mistakes, but we are better at coming up with credible explanations
as to why it is not a rationalization.
</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEisvRsgdQxSF7DocyoWa_xkIvqOy1UPNjgBynfvWll3vrcyRqP2RtvUUZ2smqxZedWe7966PmbFEIPOFdpWWc_p9-IrJ2QyQnZAORJglbNZt3R16Un3QsHdQssDgGhpXT-pGj5s/s1600/fallacies-mind.033.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEisvRsgdQxSF7DocyoWa_xkIvqOy1UPNjgBynfvWll3vrcyRqP2RtvUUZ2smqxZedWe7966PmbFEIPOFdpWWc_p9-IrJ2QyQnZAORJglbNZt3R16Un3QsHdQssDgGhpXT-pGj5s/s200/fallacies-mind.033.png" /></a></div>
<h2>Skepticism</h2>
<p>"I doubt it!" is not only a proper response to what other people say. It is
also an appropriate response to our own thought and ideas.</p>
<p>Scientific skepticism holds that science is the best way to find out things
about the world and ourselves. Scientific skeptics don't trust claims made by
people who reject science or who don't think that science is the best way to
learn about the world.</p>
<p>Scientific skeptics don't say that all extraordinary claims are false. A claim
isn't false just because it hasn't been proven true.</p>
<p>It's possible pigs can fly, but until we see the evidence we shouldn't give up
what science has taught us about pigs and flying.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgbpeeZxOHL6rrsEj1Cy9l8ZDZFjzKtQzAx-7g84tZCYxdyPzN7NvZpuQa8C-1JXOxmvA60seKqqQ2F4gDZmvDzv46xYNZKgK6Drw0PbpqHGPjGOrRF4R7fKONqTuXanMto798f/s1600/fallacies-mind.036.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgbpeeZxOHL6rrsEj1Cy9l8ZDZFjzKtQzAx-7g84tZCYxdyPzN7NvZpuQa8C-1JXOxmvA60seKqqQ2F4gDZmvDzv46xYNZKgK6Drw0PbpqHGPjGOrRF4R7fKONqTuXanMto798f/s200/fallacies-mind.036.png" /></a></div>
<h2>Meta Cognition</h2>
<p>Thinking about thinking! When you learn new facts, be aware of all the fallacies
and biases mentioned in this article. This will help prevent you from
making some mistakes.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhyE0jOtoBZg9-Q2LXsfxPwsxNmWVi084tmW0T7l4aw-XY3Xyzx_t1uy5v4h40MFwCqGHMJl7-bCfoxtmwK0iL4yYba6dMC4MJln9GtlaOJwXNsfkdgHAjsD4yIMpbhWakpU4SP/s1600/fallacies-mind.037.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhyE0jOtoBZg9-Q2LXsfxPwsxNmWVi084tmW0T7l4aw-XY3Xyzx_t1uy5v4h40MFwCqGHMJl7-bCfoxtmwK0iL4yYba6dMC4MJln9GtlaOJwXNsfkdgHAjsD4yIMpbhWakpU4SP/s200/fallacies-mind.037.png" /></a></div>
<h2>Bias Blind Spot</h2>
<p>The bias blind spot is the cognitive bias of failing to compensate for one's
own cognitive biases. Even if we know everything I've written about here, we
have a tendency to underestimate our potential for self-deception. To see
ourselves as rational beings is the greatest self-deception of all.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhtCl8bbq_bBGpqw1_CSNtLZ-aaIzgrupX-BTewDpXD1lSS08CjJs1HBhXFeMuGhJkK14MRZokB91JZFPYsPIcUODPT1eDr2IFAkjvoqi9ZSxVDYmnvUfWTQesqeBDE0anTNQ-9/s1600/fallacies-mind.038.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhtCl8bbq_bBGpqw1_CSNtLZ-aaIzgrupX-BTewDpXD1lSS08CjJs1HBhXFeMuGhJkK14MRZokB91JZFPYsPIcUODPT1eDr2IFAkjvoqi9ZSxVDYmnvUfWTQesqeBDE0anTNQ-9/s200/fallacies-mind.038.png" /></a></div>
<h2>Richard Feynman</h2>
<pre><em>
The first principle is that you must not fool yourself -
and you are the easiest person to fool.
-- Richard Feynman
</em></pre>
<h2>References</h2>
<ul>
<li><a href="http://en.wikipedia.org/wiki/List_of_cognitive_biases">Wikipedia's list of cognitive biases</a></li>
<li><a href="http://www.amazon.com/Thinking-Fast-Slow-Daniel-Kahneman/dp/0374533555?tag=thtasta-20">Thinking fast and slow</a>, Daniel Kahneman</li>
<li><a href="http://www.amazon.com/How-Know-What-Isnt-Fallibility/dp/0029117062?tag=thtasta-20">How we know what isn’t so</a>, Thomas Gilovich</li>
<li><a href="http://www.amazon.com/Fooled-Randomness-Hidden-Markets-Incerto/dp/0812975219?tag=thtasta-20">Fooled by Randomness</a>, Nassim Taleb</li>
<li><a href="http://www.amazon.com/Drunkards-Walk-Randomness-Rules-Lives-ebook/dp/B002RI9E0K?tag=thtasta-20">The Drunkards Walk</a>, Leonard Mlodinow</li>
<li><a href="http://www.amazon.com/Naked-Statistics-Stripping-Dread-Data/dp/1480590185?tag=thtasta-20">Naked Statistics</a>, Charles Wheelan</li>
<li><a href="http://www.amazon.com/The-5-Elements-Effective-Thinking/dp/0691156662?tag=thtasta-20">The Five Elements of Effective Thinking</a>, Edward B. Burger, Michael Starbird</li>
</ul>
Anders Janmyrhttp://www.blogger.com/profile/16081537018884349622noreply@blogger.com2tag:blogger.com,1999:blog-34049130.post-29382919373190084052014-05-11T08:55:00.000+02:002014-05-11T09:14:41.519+02:00Ping-Pong Pairing Over Git<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhPTBFgi16biEfCrOj7E9OT2YDpIQqFGpPoaF7aMMSr0bmo918keTy-lbw43FobUzsZrwBMGX1b782gPPcnzOFUGg0z3ZZLo3rZ4K0i7nklFk9L4B59Wy-CSfAzHuwbQsxrl__3/s1600/pingpong.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhPTBFgi16biEfCrOj7E9OT2YDpIQqFGpPoaF7aMMSr0bmo918keTy-lbw43FobUzsZrwBMGX1b782gPPcnzOFUGg0z3ZZLo3rZ4K0i7nklFk9L4B59Wy-CSfAzHuwbQsxrl__3/s200/pingpong.jpg" /></a></div>
<p>When practicing new programming techniques I am a fan of <a href="http://c2.com/cgi/wiki?PairProgrammingPingPongPattern">ping-pong pairing</a>.
Ping-pong pairing is a way of pairing with TDD that evenly distributes the
amount each programmer spends in front of the keyboard and the amount of test
code versus actual code each programmer writes.</p>
<p>The description from the C2 Wiki reads:</p>
<blockquote><pre><tt>Pair Programming Ping Pong Pattern
* A writes a new test and sees that it fails.
* B implements the code needed to pass the test.
* B writes the next test and sees that it fails.
* A implements the code needed to pass the test.
And so on. Refactoring is done whenever the need arises by whoever is driving.
</tt></pre></blockquote>
<p>Two programmers sit in from of <em>one</em> computer, with <em>one</em> keyboard and <em>one</em>
screen.</p>
<p>Ping-pong pairing is great in a learning context because it keeps both
programmers in front of the keyboard and it encourages conversation.</p>
<p>But, there are a couple of problems. Keyboards and editors! What if one
programmer uses Dvorak and the other Qwerty? Or, one programmer cannot even
think of writing another line of code without <em>Das Keyboard</em> while the other
prefers an ergonomic keyboard? Or, one programmer uses Vim on OSX and the other
Notepad on Windows?</p>
<h2>Ping-Pong Pairing Over Git</h2>
<p>What if we alter the setup to give each user their own computer, keyboard,
screen, rubber duck, or whatever tickles their fancy? It would seem that this
isn't paring any more! But, if we place the pair side-by-side and let them
communicate over Git, we actually get a very nice flow. There is still only one
person typing on the keyboard at a time, but they are typing on their own
keyboard.</p>
<p>The pattern above is changed to:</p>
<blockquote><pre><tt>Ping Pong Paring Over Git Pattern
* A writes a new test and sees that it fails.
* A commits the failing test and pushes the code to Git
* B pulls the code from Git.
* B implements the code needed to pass the test.
* B writes the next test and sees that it fails.
* B commits the failing test and pushes the code to Git
* A pulls the code from Git.
* A implements the code needed to pass the test.
And so on.
</tt></pre></blockquote>
<p>I've tried this pattern on a couple of code retreats and it is actually pretty
smooth. To make it even more smooth I implemented a simple command line
utility, <a href="https://github.com/andersjanmyr/tapir">tapir</a> that allows for simple
communication between the two computers and automates the pulling of the new
code. It works like this. Each programmer starts a listener on their machine
that pulls the code when it receives a message.</p>
<blockquote><pre><tt><i><font color="#9A1900"># Start a listener on 'my-topic', run git pull when a message arrives</font></i>
tapir --cmd listen --script <font color="#FF0000">'git pull'</font> mytopic
</tt></pre></blockquote>
<p>Write a simple script to combine <code>git push</code> with a calling <code>tapir mytopic</code></p>
<blockquote><pre><tt><i><font color="#9A1900">#!/bin/sh</font></i>
<i><font color="#9A1900"># ./push script pushes code to git and pings the tapir-server</font></i>
git push
tapir mytopic
</tt></pre></blockquote>
<p>Now, instead of calling <code>git push</code>, you call <code>./push</code> and the code is
automatically pulled on the other machine, eliminating one step from the loop.</p>
<h2>Summary</h2>
<p>Ping-pong pairing over Git is nice! If you are interested in trying it out
I have a <a href="https://github.com/andersjanmyr/roman-numerals-kata">roman numerals kata</a> with setup code for
multiple languages, currently: Clojure, ClojureScript, Javascript, Lua,
VimScript, Objective-C, PHP, Ruby and C.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiUafQ354hulpN52G3jYSvSiPlhvJqDWivajDknUZ_xuzlFCj8NuCDM8f9_ymOZnJ_DfUMf8wNzkr_WrG8v48pVlbcv52q8aZS6N5YqsQg2Vt6Hjz3Rzv82edBLRzxn4fT7TLld/s1600/Tapir+3.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiUafQ354hulpN52G3jYSvSiPlhvJqDWivajDknUZ_xuzlFCj8NuCDM8f9_ymOZnJ_DfUMf8wNzkr_WrG8v48pVlbcv52q8aZS6N5YqsQg2Vt6Hjz3Rzv82edBLRzxn4fT7TLld/s200/Tapir+3.jpg" /></a></div>
<p>The <a href="https://github.com/andersjanmyr/tapir">tapir</a> command line utility is also
pretty interesting as it uses ServerSentEvents to communicate over a standard
http server, <a href="http://tapir-server.herokuapp.com/">http://tapir-server.herokuapp.com/</a></p>
<p>Why is the utility called <code>tapir</code>? Because, pingpong and ping-pong were already
taken and I like tapirs! :).</p>
Anders Janmyrhttp://www.blogger.com/profile/16081537018884349622noreply@blogger.com0tag:blogger.com,1999:blog-34049130.post-89207752832888516502014-03-25T21:09:00.001+01:002014-03-31T09:11:23.338+02:00Running Scripts with npm<div class="separator" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;">
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiR9cAaH5aVa9l0mUhgSZjUoWcEnfnenoFGFlJcqCwrdyLm3vzmwBhH_1KanXlAGrdwAtJ1RSBf01y6vG6XsddrMGo3YDpkm6HVZ7A7HWHhzP_7xwCDsUyiv9S_R4ZRX42YCB3X/s1600/npm-script.png" /></div>
<p>Most people are aware that is is possible to define scripts in <code>package.json</code>
which can be run with <code>npm start</code> or <code>npm test</code>, but <code>npm</code> scripts can do a lot
more than simply start servers and run tests.</p>
<p>Here is a typical <code>package.json</code> configuration.</p>
<!-- Generator: GNU source-highlight 3.1.7
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt><i><font color="#9A1900">// package.json</font></i>
<i><font color="#9A1900">// Define start and test targets</font></i>
<font color="#FF0000">{</font>
<font color="#FF0000">"name"</font><font color="#990000">:</font> <font color="#FF0000">"death-clock"</font><font color="#990000">,</font>
<font color="#FF0000">"version"</font><font color="#990000">:</font> <font color="#FF0000">"1.0.0"</font><font color="#990000">,</font>
<font color="#FF0000">"scripts"</font><font color="#990000">:</font> <font color="#FF0000">{</font>
<font color="#FF0000">"start"</font><font color="#990000">:</font> <font color="#FF0000">"node server.js"</font><font color="#990000">,</font>
<font color="#FF0000">"test"</font><font color="#990000">:</font> <font color="#FF0000">"mocha --reporter spec test"</font>
<font color="#FF0000">}</font><font color="#990000">,</font>
<font color="#FF0000">"devDependencies"</font><font color="#990000">:</font> <font color="#FF0000">{</font>
<font color="#FF0000">"mocha"</font><font color="#990000">:</font> <font color="#FF0000">"^1.17.1"</font>
<font color="#FF0000">}</font>
<font color="#FF0000">}</font>
<i><font color="#9A1900">// I am using comments in JSON files for clarity.</font></i>
<i><font color="#9A1900">// Comments won't work in real JSON files.</font></i>
</tt></pre>
<p><code>start</code>, actually defaults to <code>node server.js</code>, so the above declaration is
redundant. In order for the test command to work with <code>mocha</code>, I also need to
include it in the <code>devDependencies</code> section (it works in the <code>dependencies</code>
section also, but since it is not needed in production it is better to declare
it here).</p>
<p>The reason the above test command, <code>mocha --reporter spec test</code>, works is
because <code>npm</code> looks for binaries inside <code>node_modules/.bin</code> and when <code>mocha</code> was
installed it installed <code>mocha</code> into this directory.</p>
<p>The code that describes what will be installed into the <code>bin</code> directory is
defined in <code>mocha</code>'s
<a href="https://github.com/visionmedia/mocha/blob/master/package.json">package.json</a>
and it looks like this:</p>
<!-- Generator: GNU source-highlight 3.1.7
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt><i><font color="#9A1900">// Macha package.json</font></i>
<font color="#FF0000">{</font>
<font color="#FF0000">"name"</font><font color="#990000">:</font> <font color="#FF0000">"mocha"</font><font color="#990000">,</font>
<font color="#990000">...</font>
<font color="#FF0000">"bin"</font><font color="#990000">:</font> <font color="#FF0000">{</font>
<font color="#FF0000">"mocha"</font><font color="#990000">:</font> <font color="#FF0000">"./bin/mocha"</font><font color="#990000">,</font>
<font color="#FF0000">"_mocha"</font><font color="#990000">:</font> <font color="#FF0000">"./bin/_mocha"</font>
<font color="#FF0000">}</font><font color="#990000">,</font>
<font color="#990000">...</font>
<font color="#FF0000">}</font>
</tt></pre>
<p>As we can see in the above declaration, <code>mocha</code> has two binaries, <code>mocha</code> and
<code>_mocha</code>.</p>
<p>Many packages have a <code>bin</code> section, declaring scripts that can be called from
<code>npm</code> similar to <code>mocha</code>. To find out what binaries we have in our project we
can run <code>ls node_modules/.bin</code></p>
<!-- Generator: GNU source-highlight 3.1.7
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt><i><font color="#9A1900"># Scripts availble in one of my projects</font></i>
$ ls node_modules<font color="#990000">/.</font>bin
_mocha browserify envify jshint
jsx lessc lesswatcher mocha
nodemon uglifyjs watchify
</tt></pre>
<h2>Invoking Commands</h2>
<p>Both <code>start</code> and <code>test</code> are special values and can be invoked directly.</p>
<!-- Generator: GNU source-highlight 3.1.7
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt><i><font color="#9A1900"># Run script declared by "start"</font></i>
$ npm start
$ npm run start
<i><font color="#9A1900"># Run script declared by "test"</font></i>
$ npm <b><font color="#0000FF">test</font></b>
$ npm run <b><font color="#0000FF">test</font></b>
</tt></pre>
<p>All other values will have to be invoked by <code>npm run</code>. <code>npm run</code> is actually a
shortcut of <code>npm run-script</code>.</p>
<!-- Generator: GNU source-highlight 3.1.7
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt><font color="#FF0000">{</font>
<font color="#990000">...</font>
<font color="#FF0000">"scripts"</font><font color="#990000">:</font> <font color="#FF0000">{</font>
<i><font color="#9A1900">// watch-test starts a mocha watcher that listens for changes</font></i>
<font color="#FF0000">"watch-test"</font><font color="#990000">:</font> <font color="#FF0000">"mocha --watch --reporter spec test"</font>
<font color="#FF0000">}</font><font color="#990000">,</font>
<font color="#FF0000">}</font>
</tt></pre>
<p>The above code must be invoked with <code>npm run watch-test</code>, <em>npm watch-test</em> will
fail.</p>
<h3>Running Binaries Directly</h3>
<p>All the above examples consists of running scripts that are declared in
<code>package.json</code> but this is not required. Any of the commands in
<code>node_modules/.bin</code> can be invoked with <code>npm run</code>. This means that I can invoke
<code>mocha</code> by running <code>npm run mocha</code> directly instead of running it with <code>mocha
test</code>.</p>
<h3>Code Completion</h3>
<p>With a lot of modules providing commands it can be difficult to remember what
all of them are. Wouldn't it be nice if we could have some command completion
to help us out? It turns out we can! <code>npm</code> follows the superb practice of
providing its own command completion. By running the command <code>npm completion</code>
we get a completion script that we can source to get completion for all the
normal <code>npm</code> commands including completion for <code>npm run</code>. Awesome!</p>
<p>I usually put each of my completion script into their own file which I invoke
from <code>.bashrc</code>.</p>
<!-- Generator: GNU source-highlight 3.1.7
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt><i><font color="#9A1900"># npm_completion.sh</font></i>
<font color="#990000">.</font> <font color="#990000"><(</font>npm completion<font color="#990000">)</font>
<i><font color="#9A1900"># Some output from one of my projects</font></i>
$ npm run <font color="#990000"><</font>tab<font color="#990000">></font>
nodemon browserify build
build-js build-less start
jshint <b><font color="#0000FF">test</font></b> deploy
less uglify-js express
mocha watch watch-js
watch-less watch-server
</tt></pre>
<p>Pretty cool!</p>
<h2>Combining Commands</h2>
<p>The above features gets us a long way but sometimes we want to do more than one
thing at a time. It turns out that <code>npm</code> supports this too. <code>npm</code> runs the
scripts by passing the line to <code>sh</code>. This allows us to combine commands just as
we can do on the command line.</p>
<h3>Piping</h3>
<p>Lets say that I want to use <code>browserify</code> to pack my Javascript files into a bundle
and then I want to minify the bundle with <code>uglifyjs</code>. I can do this by piping
(|) the output from <code>browserify</code> into <code>uglifyjs</code>. Simple as pie!</p>
<!-- Generator: GNU source-highlight 3.1.7
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt> <i><font color="#9A1900">//package.json</font></i>
<i><font color="#9A1900">// Reactify tells browserify to handle facebooks extended React syntax</font></i>
<font color="#FF0000">"scripts"</font><font color="#990000">:</font> <font color="#FF0000">{</font>
<font color="#FF0000">"build-js"</font><font color="#990000">:</font> <font color="#FF0000">"browserify -t reactify app/js/main.js | uglifyjs -mc > static/bundle.js"</font>
<font color="#FF0000">}</font><font color="#990000">,</font>
<i><font color="#9A1900">// Added the needed dependencies</font></i>
<font color="#FF0000">"devDependencies"</font><font color="#990000">:</font> <font color="#FF0000">{</font>
<font color="#FF0000">"browserify"</font><font color="#990000">:</font> <font color="#FF0000">"^3.14.0"</font><font color="#990000">,</font>
<font color="#FF0000">"reactify"</font><font color="#990000">:</font> <font color="#FF0000">"^0.5.1"</font><font color="#990000">,</font>
<font color="#FF0000">"uglify-js"</font><font color="#990000">:</font> <font color="#FF0000">"^2.4.8"</font>
<font color="#FF0000">}</font>
</tt></pre>
<h3>Anding</h3>
<p>Another use case for running commands is to run a command only if the previous
command is successful. This is easily done with <code>and</code> (&&). Or (||), naturally,
also works.</p>
<!-- Generator: GNU source-highlight 3.1.7
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt> <font color="#FF0000">"scripts"</font><font color="#990000">:</font> <font color="#FF0000">{</font>
<i><font color="#9A1900">// Run build-less if build-less succeeds</font></i>
<font color="#FF0000">"build"</font><font color="#990000">:</font> <font color="#FF0000">"npm run build-js && npm run build-less"</font><font color="#990000">,</font>
<font color="#990000">...</font>
<font color="#FF0000">"build-js"</font><font color="#990000">:</font> <font color="#FF0000">"browserify -t reactify app/js/main.js | uglifyjs -mc > static/bundle.js"</font><font color="#990000">,</font>
<font color="#FF0000">"build-less"</font><font color="#990000">:</font> <font color="#FF0000">"lessc app/less/main.less static/main.css"</font>
<font color="#FF0000">}</font>
</tt></pre>
<p>Here I run two scripts declared in my <code>package.json</code> in combination with the
command <code>build</code>. Running scripts from other scripts is different from running
binaries, they have to prefixed with <code>npm run</code>.</p>
<h3>Concurrent</h3>
<p>Sometimes it is also nice to be able to run multiple commands at the
concurrently. This is easily done by using <code>&amp;</code> to run them as background jobs.</p>
<!-- Generator: GNU source-highlight 3.1.7
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt> <font color="#FF0000">"scripts"</font><font color="#990000">:</font> <font color="#FF0000">{</font>
<i><font color="#9A1900">// Run watch-js, watch-less and watch-server concurrently</font></i>
<font color="#FF0000">"watch"</font><font color="#990000">:</font> <font color="#FF0000">"npm run watch-js & npm run watch-less & npm run watch-server"</font><font color="#990000">,</font>
<font color="#FF0000">"watch-js"</font><font color="#990000">:</font> <font color="#FF0000">"watchify app/js/main.js -t reactify -o static/bundle.js -dv"</font><font color="#990000">,</font>
<font color="#FF0000">"watch-less"</font><font color="#990000">:</font> <font color="#FF0000">"nodemon --watch app/less/*.less --ext less --exec 'npm run build-less'"</font><font color="#990000">,</font>
<font color="#FF0000">"watch-server"</font><font color="#990000">:</font> <font color="#FF0000">"nodemon --ignore app --ignore static server.js"</font>
<font color="#FF0000">}</font><font color="#990000">,</font>
<i><font color="#9A1900">// Add required dependencies</font></i>
<font color="#FF0000">"devDependencies"</font><font color="#990000">:</font> <font color="#FF0000">{</font>
<font color="#FF0000">"watchify"</font><font color="#990000">:</font> <font color="#FF0000">"^0.6.2"</font><font color="#990000">,</font>
<font color="#FF0000">"nodemon"</font><font color="#990000">:</font> <font color="#FF0000">"^1.0.15"</font>
<font color="#FF0000">}</font>
</tt></pre>
<p>The above scripts contain a few interesting things. First of all <code>watch</code> uses
<code>&amp;</code> to run three watch jobs concurrently. When the command is killed, by
pressing <code>Ctrl-C</code>, all the jobs are killed, since they are all run with the
same parent process.</p>
<p><code>watchify</code> is a way to run <code>browserify</code> in watch mode. <code>watch-server</code> uses
<code>nodemon</code> in the standard way and restarts the server whenever a relevant file
has changed.</p>
<p><code>watch-less</code> users <code>nodemon</code> in a less well-known way. It runs a script when
any of the less-files changes and compiles them into CSS by running
<code>npm run build-less</code>. Please note that the option <code>--ext less</code> is required for
this to work. <code>--exec</code> is the option that allows <code>nodemon</code> to run external
commands.</p>
<h3>Complex Scripts</h3>
<p>For more complex scripts I prefer to write them in Bash, but I usually include
a declaration in <code>package.json</code> to run the command. Here, for example, is a
small script that deploys the compiled assets to Heroku by adding them to a
deploy branch and pushing that branch to Heroku.</p>
<!-- Generator: GNU source-highlight 3.1.7
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt><i><font color="#9A1900">#!/bin/bash</font></i>
<b><font color="#0000FF">set</font></b> -o errexit <i><font color="#9A1900"># Exit on error</font></i>
git stash save -u <font color="#FF0000">'Before deploy'</font> <i><font color="#9A1900"># Stash all changes, including untracked files, before deploy</font></i>
git checkout deploy
git merge master --no-edit <i><font color="#9A1900"># Merge in the master branch without prompting</font></i>
npm run build <i><font color="#9A1900"># Generate the bundled Javascript and CSS</font></i>
<b><font color="#0000FF">if</font></b> <font color="#009900">$(</font>git commit -am Deploy<font color="#990000">);</font> <b><font color="#0000FF">then</font></b> <i><font color="#9A1900"># Commit the changes, if any</font></i>
echo <font color="#FF0000">'Changes Committed'</font>
<b><font color="#0000FF">fi</font></b>
git push heroku deploy<font color="#990000">:</font>master <i><font color="#9A1900"># Deploy to Heroku</font></i>
git checkout master <i><font color="#9A1900"># Checkout master again</font></i>
git stash pop <i><font color="#9A1900"># And restore the changes</font></i>
</tt></pre>
<p>Add the script to <code>package.json</code> so that it can be run with <code>npm run deploy</code>.</p>
<!-- Generator: GNU source-highlight 3.1.7
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt> <font color="#FF0000">"scripts"</font><font color="#990000">:</font> <font color="#FF0000">{</font>
<font color="#FF0000">"deploy"</font><font color="#990000">:</font> <font color="#FF0000">"./bin/deploy.sh"</font>
<font color="#FF0000">}</font><font color="#990000">,</font>
</tt></pre>
<h2>Conclusion</h2>
<p><code>npm</code> is a lot more than a package manager for Node. By configuring it properly
I can handle most of my scripting needs.</p>
<p>Configuring <code>start</code> and <code>test</code> also sets me up for integration with SaaS
providers such as Heroku and TravisCI. Another good reason to do it.</p>Anders Janmyrhttp://www.blogger.com/profile/16081537018884349622noreply@blogger.com9tag:blogger.com,1999:blog-34049130.post-47126845019229870732014-01-19T10:56:00.000+01:002014-06-24T14:50:05.492+02:00Clean Grunt<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgb8xx_3Yn4of4G2AbgukarzhBxDaXjgH2dFm_mLVaHMuJrbXW0Y_g4fTUWtUCWQhhyphenhyphendB1KZPi0PTYic3qMPFDQaFhRM5Hfl02QY-u2_takVL4Rp7Qs7WfNTZC4hQ97Qlm2_Zvj/s1600/cleanpig.gif" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgb8xx_3Yn4of4G2AbgukarzhBxDaXjgH2dFm_mLVaHMuJrbXW0Y_g4fTUWtUCWQhhyphenhyphendB1KZPi0PTYic3qMPFDQaFhRM5Hfl02QY-u2_takVL4Rp7Qs7WfNTZC4hQ97Qlm2_Zvj/s320/cleanpig.gif" /></a></div>
<p>
Grunt is the tool of choice for many client side web projects. But, often the
gruntfiles look like a mess. I believe the reason for this is that many people
don't care about keeping it clean.
</p>
<p>
On top of that, the file is often generated
by a tool, such as Yeoman, and not cleaned up after. I happen to think that
the gruntfile should be clean and here is a how to do it.
</p>
<p>
Here is how the project structure looks in development mode. I keep all my
client side code in an <code>app</code> directory and I use Bower to install external
components into <code>app/components</code>
</p>
<br class='clear'/>
<blockquote><pre><tt>app
components
jquery
jquery<font color="#990000">.</font>js
momentjs
moment<font color="#990000">.</font>js
images
bower-logo<font color="#990000">.</font>png
grunt-logo<font color="#990000">.</font>svg
index<font color="#990000">.</font>html
scripts
main<font color="#990000">.</font>js
model<font color="#990000">.</font>js
view<font color="#990000">.</font>js
styles
images<font color="#990000">.</font>css
main<font color="#990000">.</font>css
</tt></pre></blockquote>
<p>I will use <code>less, watch, concat, uglify, filerev</code> and <code>usemin</code> to
optimize it and turn it into this.</p>
<blockquote><pre><tt>dist
app
images
bower-logo<font color="#990000">.</font>fd05710aa2cb9502dc90<font color="#990000">.</font>png
grunt-logo<font color="#990000">.</font>16c32bb187681923d5a7<font color="#990000">.</font>svg
index<font color="#990000">.</font>html
scripts
main<font color="#990000">.</font>359737238b7dc0972e52<font color="#990000">.</font>js
styles
main<font color="#990000">.</font>6873d02f25d2385b9ec8<font color="#990000">.</font>css
</tt></pre></blockquote>
<p>The above structure is good because it serves <em>one</em> CSS file, <em>one</em> Javascript
file, and everything apart from <code>index.html</code> is named with an MD5 checksum that
allow me to cache everything infinitely!</p>
<h2>Loading External Tasks</h2>
<p>Loading tasks in Grunt is done with <code>grunt.loadNpmTasks</code> but since all
dependenciies is already declared in <code>package.json</code> there is no need to name
them again. So instead we use <code>matchdep</code> to load all Grunt dependencies
automatically.</p>
<blockquote><pre><tt><i><font color="#9A1900">// Load all files starting with `grunt-`</font></i>
<b><font color="#000000">require</font></b><font color="#990000">(</font><font color="#FF0000">'matchdep'</font><font color="#990000">).</font><b><font color="#000000">filterDev</font></b><font color="#990000">(</font><font color="#FF0000">'grunt-*'</font><font color="#990000">).</font><b><font color="#000000">forEach</font></b><font color="#990000">(</font>grunt<font color="#990000">.</font>loadNpmTasks<font color="#990000">);</font>
</tt></pre></blockquote>
<p>The relevant section in <code>package.json</code> contains these files. All Grunt plugins
follow the <code>grunt-</code> naming convention.</p>
<blockquote><pre><tt> <font color="#FF0000">"devDependencies"</font><font color="#990000">:</font> <font color="#FF0000">{</font>
<font color="#FF0000">"bower"</font><font color="#990000">:</font> <font color="#FF0000">"~1.2.8"</font><font color="#990000">,</font>
<font color="#FF0000">"grunt"</font><font color="#990000">:</font> <font color="#FF0000">"~0.4.2"</font><font color="#990000">,</font>
<font color="#FF0000">"matchdep"</font><font color="#990000">:</font> <font color="#FF0000">""</font><font color="#990000">,</font>
<font color="#FF0000">"grunt-contrib-jshint"</font><font color="#990000">:</font> <font color="#FF0000">""</font><font color="#990000">,</font>
<font color="#FF0000">"grunt-contrib-less"</font><font color="#990000">:</font> <font color="#FF0000">""</font><font color="#990000">,</font>
<font color="#FF0000">"grunt-contrib-copy"</font><font color="#990000">:</font> <font color="#FF0000">""</font><font color="#990000">,</font>
<font color="#FF0000">"grunt-contrib-clean"</font><font color="#990000">:</font> <font color="#FF0000">""</font><font color="#990000">,</font>
<font color="#FF0000">"grunt-contrib-watch"</font><font color="#990000">:</font> <font color="#FF0000">""</font><font color="#990000">,</font>
<font color="#FF0000">"grunt-express-server"</font><font color="#990000">:</font> <font color="#FF0000">""</font><font color="#990000">,</font>
<font color="#FF0000">"grunt-contrib-cssmin"</font><font color="#990000">:</font> <font color="#FF0000">""</font><font color="#990000">,</font>
<font color="#FF0000">"grunt-usemin"</font><font color="#990000">:</font> <font color="#FF0000">""</font><font color="#990000">,</font>
<font color="#FF0000">"grunt-filerev"</font><font color="#990000">:</font> <font color="#FF0000">""</font><font color="#990000">,</font>
<font color="#FF0000">"grunt-contrib-concat"</font><font color="#990000">:</font> <font color="#FF0000">""</font><font color="#990000">,</font>
<font color="#FF0000">"grunt-contrib-uglify"</font><font color="#990000">:</font> <font color="#FF0000">""</font>
<font color="#FF0000">}</font>
<font color="#FF0000">}</font>
</tt></pre></blockquote>
<h2>JsHint</h2>
<p>I think it is a good idea to run JsHint for all my files including the
Gruntfile and here is how I configure it.</p>
<blockquote><pre><tt><i><font color="#9A1900">// JsHint configuration is read from packages.json</font></i>
<b><font color="#0000FF">var</font></b> pkg <font color="#990000">=</font> grunt<font color="#990000">.</font>file<font color="#990000">.</font><b><font color="#000000">readJSON</font></b><font color="#990000">(</font><font color="#FF0000">'package.json'</font><font color="#990000">);</font>
grunt<font color="#990000">.</font><b><font color="#000000">initConfig</font></b><font color="#990000">(</font><font color="#FF0000">{</font>
pkg<font color="#990000">:</font> pkg<font color="#990000">,</font>
<i><font color="#9A1900">// JsHint</font></i>
jshint<font color="#990000">:</font> <font color="#FF0000">{</font>
options<font color="#990000">:</font> pkg<font color="#990000">.</font>jshintConfig<font color="#990000">,</font>
all<font color="#990000">:</font> <font color="#990000">[</font>
<font color="#FF0000">'Gruntfile.js'</font><font color="#990000">,</font>
<font color="#FF0000">'app/scripts/**/*.js'</font><font color="#990000">,</font>
<font color="#FF0000">'test/**/*.js'</font>
<font color="#990000">]</font>
<font color="#FF0000">}</font>
<font color="#FF0000">}</font>
</tt></pre></blockquote>
<p>Newer versions of JsHint can pick up configuration from <code>package.json</code> and I
take advantage of this so I don't have a duplicate configuration in a <code>.jshint</code>
file that is normally added when using a generated project.</p>
<p>The relevant section in <code>package.json</code> is defined like this:</p>
<blockquote><pre><tt> <font color="#FF0000">"jshintConfig"</font><font color="#990000">:</font> <font color="#FF0000">{</font>
<font color="#990000">...</font>
<font color="#FF0000">"maxparams"</font><font color="#990000">:</font> <font color="#993399">4</font><font color="#990000">,</font>
<font color="#FF0000">"maxdepth"</font><font color="#990000">:</font> <font color="#993399">2</font><font color="#990000">,</font>
<font color="#FF0000">"maxcomplexity"</font><font color="#990000">:</font> <font color="#993399">6</font><font color="#990000">,</font>
<font color="#990000">...</font>
<font color="#FF0000">}</font>
</tt></pre></blockquote>
<p>I truncated the section for brevity but I kept my favorite configuration
options that deal with complexity and forces me to keep my code simple.</p>
<h2>Less</h2>
<p>As I wrote in <a href="http://anders.janmyr.com/2012/08/css-good-practices.html">CSS Good Practices</a>,
I think using a CSS preprocessor is a really good idea and I use Less in this
project. Since Less is a superset of CSS all I have to do to use less is to
change the extension from <code>.css</code> to <code>.less</code> and configure Grunt to convert Less
files into CSS. In development mode I like to have the CSS files in the same
place I would have put them if I wasn't using Less. To avoid accidentally
checking the generated files into source control I add the following line to
<code>.gitignore</code></p>
<blockquote><pre><tt><i><font color="#9A1900"># .gitignore</font></i>
app/styles<font color="#990000">/*.</font>css
</tt></pre></blockquote>
<p>Here is the configuration for generating a CSS file. I add two targets, one for
development and one for release which is compressed.</p>
<blockquote><pre><tt><i><font color="#9A1900">// Less</font></i>
less<font color="#990000">:</font> <font color="#FF0000">{</font>
dev<font color="#990000">:</font> <font color="#FF0000">{</font>
src<font color="#990000">:</font> <font color="#FF0000">'app/styles/main.css'</font><font color="#990000">,</font>
dest<font color="#990000">:</font> <font color="#FF0000">'app/styles/main.less'</font>
<font color="#FF0000">}</font><font color="#990000">,</font>
release<font color="#990000">:</font> <font color="#FF0000">{</font>
src<font color="#990000">:</font> <font color="#FF0000">'app/styles/main.css'</font><font color="#990000">,</font>
dest<font color="#990000">:</font> <font color="#FF0000">'dist/app/styles/main.less'</font><font color="#990000">,</font>
options<font color="#990000">:</font> <font color="#FF0000">{</font>
compress<font color="#990000">:</font> <b><font color="#0000FF">true</font></b>
<font color="#FF0000">}</font>
<font color="#FF0000">}</font>
<font color="#FF0000">}</font>
</tt></pre></blockquote>
<p>As you can see I only name one less file. I think it is a good idea to include
all less files via import statements.</p>
<blockquote><pre><tt><i><font color="#9A1900">// Less files are automatically included and don't generate new requests.</font></i>
@import 'other<font color="#990000">-</font>less<font color="#990000">-</font>file<font color="#993399">.less</font>'<font color="#990000">;</font>
</tt></pre></blockquote>
<h2>Watch</h2>
<p>In development mode I also like to have a file watcher that generates the CSS
files automatically when I change a less file. Here is the configuration.</p>
<blockquote><pre><tt><i><font color="#9A1900">// Watch</font></i>
watch<font color="#990000">:</font> <font color="#FF0000">{</font>
<i><font color="#9A1900">// watch:less invokes less:dev when less files change</font></i>
less<font color="#990000">:</font> <font color="#FF0000">{</font>
files<font color="#990000">:</font> <font color="#990000">[</font><font color="#FF0000">'app/styles/*.less'</font><font color="#990000">],</font>
tasks<font color="#990000">:</font> <font color="#990000">[</font><font color="#FF0000">'less:dev'</font><font color="#990000">]</font>
<font color="#FF0000">}</font>
<font color="#FF0000">}</font>
</tt></pre></blockquote>
<h2>Clean</h2>
<p>It is also a good idea to be able to remove generated files with one command
<code>clean</code> will do that for me.</p>
<blockquote><pre><tt><i><font color="#9A1900">// Clean</font></i>
clean<font color="#990000">:</font> <font color="#FF0000">{</font>
<i><font color="#9A1900">// clean:release removes generated files</font></i>
release<font color="#990000">:</font> <font color="#990000">[</font>
<font color="#FF0000">'dist'</font><font color="#990000">,</font>
<font color="#FF0000">'app/styles/*.css'</font>
<font color="#990000">]</font>
<font color="#FF0000">}</font>
</tt></pre></blockquote>
<h2>Concat, Uglify and Usemin Prepare</h2>
<p>To concatenate and minify the Javascript files, I use <code>concat</code> and <code>uglify</code>.
But I don't want the files used in <code>index.html</code> to be automatically included.
To do this I need to use <code>useminPrepare</code>. It is one of two tasks included in
<code>grunt-usemin</code>, the other is unsuprisingly called <code>usemin</code> and I will describe
it later.</p>
<p><code>useminPrepare</code> parses HTML files, looking for tags that follow a distinct
pattern, <code>&lt;!-- build:js outputfile.js --&gt;</code> and extracts the filenames from
script tags. These files are then injected into the <code>concat</code> and <code>uglify</code>
tasks. So, there is no need to provide a configuration for those tasks.</p>
<blockquote><pre><tt><i><font color="#9A1900"><!-- app/index.html --></font></i>
<i><font color="#9A1900"><!-- build:js scripts/main.js --></font></i>
<b><font color="#0000FF"><script</font></b> <font color="#009900">src</font><font color="#990000">=</font><font color="#FF0000">"components/jquery/jquery.js"</font> <font color="#009900">defer</font><b><font color="#0000FF">></script></font></b>
<b><font color="#0000FF"><script</font></b> <font color="#009900">src</font><font color="#990000">=</font><font color="#FF0000">"components/momentjs/moment.js"</font> <font color="#009900">defer</font><b><font color="#0000FF">></script></font></b>
<b><font color="#0000FF"><script</font></b> <font color="#009900">src</font><font color="#990000">=</font><font color="#FF0000">"scripts/model.js"</font> <font color="#009900">defer</font><b><font color="#0000FF">></script></font></b>
<b><font color="#0000FF"><script</font></b> <font color="#009900">src</font><font color="#990000">=</font><font color="#FF0000">"scripts/view.js"</font> <font color="#009900">defer</font><b><font color="#0000FF">></script></font></b>
<b><font color="#0000FF"><script</font></b> <font color="#009900">src</font><font color="#990000">=</font><font color="#FF0000">"scripts/main.js"</font> <font color="#009900">defer</font><b><font color="#0000FF">></script></font></b>
<i><font color="#9A1900"><!-- endbuild --></font></i>
</tt></pre></blockquote>
<blockquote><pre><tt><i><font color="#9A1900">/// userminPrepare</font></i>
useminPrepare<font color="#990000">:</font> <font color="#FF0000">{</font>
html<font color="#990000">:</font> <font color="#FF0000">'app/index.html'</font><font color="#990000">,</font>
options<font color="#990000">:</font> <font color="#FF0000">{</font>
dest<font color="#990000">:</font> <font color="#FF0000">'dist/app'</font>
<font color="#FF0000">}</font>
<font color="#FF0000">}</font><font color="#990000">,</font>
<i><font color="#9A1900">// Concat</font></i>
concat<font color="#990000">:</font> <font color="#FF0000">{</font>
options<font color="#990000">:</font> <font color="#FF0000">{</font>
separator<font color="#990000">:</font> <font color="#FF0000">';'</font>
<font color="#FF0000">}</font><font color="#990000">,</font>
<i><font color="#9A1900">// dist configuration is provided by useminPrepare</font></i>
dist<font color="#990000">:</font> <font color="#FF0000">{}</font>
<font color="#FF0000">}</font><font color="#990000">,</font>
<i><font color="#9A1900">// Uglify</font></i>
uglify<font color="#990000">:</font> <font color="#FF0000">{</font>
<i><font color="#9A1900">// dist configuration is provided by useminPrepare</font></i>
dist<font color="#990000">:</font> <font color="#FF0000">{}</font>
<font color="#FF0000">}</font>
</tt></pre></blockquote>
<p>There are a few things that are noteworthy above. <code>useminPrepare.options.dest</code>
works in conjunction with the value defined in the <code>build:js</code> comment in the
html file. I always designate the root directory of the generated code in the
Gruntfile and I keep the relative path to the file in the HTML file. I do this
because this configuration is reused by the <code>usemin</code> task later and configuring
it this way in <code>useminPrepare</code> keeps it simpler later.</p>
<p>Also note that <code>concat</code> and <code>uglify</code> needs to have an empty <code>dist</code> property.
Otherwise, <code>useminPrepare</code> cannot inject configuration into it.</p>
<p>Running <code>grunt useminPrepare</code> shows the generated configuration.</p>
<blockquote><pre><tt>concat<font color="#990000">:</font>
<font color="#FF0000">{</font> options<font color="#990000">:</font> <font color="#FF0000">{</font> separator<font color="#990000">:</font> <font color="#FF0000">';'</font> <font color="#FF0000">}</font><font color="#990000">,</font>
dist<font color="#990000">:</font> <font color="#FF0000">{}</font><font color="#990000">,</font>
generated<font color="#990000">:</font>
<font color="#FF0000">{</font> files<font color="#990000">:</font>
<font color="#990000">[</font> <font color="#FF0000">{</font> dest<font color="#990000">:</font> <font color="#FF0000">'.tmp/concat/scripts/main.js'</font><font color="#990000">,</font>
src<font color="#990000">:</font>
<font color="#990000">[</font> <font color="#FF0000">'app/components/momentjs/moment.js'</font><font color="#990000">,</font>
<font color="#FF0000">'app/components/jquery/jquery.js'</font><font color="#990000">,</font>
<font color="#FF0000">'app/scripts/model.js'</font><font color="#990000">,</font>
<font color="#FF0000">'app/scripts/view.js'</font><font color="#990000">,</font>
<font color="#FF0000">'app/scripts/main.js'</font> <font color="#990000">]</font> <font color="#FF0000">}</font> <font color="#990000">]</font> <font color="#FF0000">}</font> <font color="#FF0000">}</font>
uglify<font color="#990000">:</font>
dist<font color="#990000">:</font> <font color="#FF0000">{}</font><font color="#990000">,</font>
generated<font color="#990000">:</font>
<font color="#FF0000">{</font> files<font color="#990000">:</font>
<font color="#990000">[</font> <font color="#FF0000">{</font> dest<font color="#990000">:</font> <font color="#FF0000">'dist/app/scripts/main.js'</font><font color="#990000">,</font>
src<font color="#990000">:</font> <font color="#990000">[</font> <font color="#FF0000">'.tmp/concat/scripts/main.js'</font> <font color="#990000">]</font> <font color="#FF0000">}</font> <font color="#990000">]</font> <font color="#FF0000">}</font> <font color="#FF0000">}</font>
</tt></pre></blockquote>
<p>Alright, now we have minified both CSS and Javascript, it is time to move the
files that don't need minification, images and html files.</p>
<h2>Copy</h2>
<blockquote><pre><tt><i><font color="#9A1900">// Copy HTML and fonts</font></i>
copy<font color="#990000">:</font> <font color="#FF0000">{</font>
<i><font color="#9A1900">// copy:release copies all html and image files to dist</font></i>
<i><font color="#9A1900">// preserving the structure</font></i>
release<font color="#990000">:</font> <font color="#FF0000">{</font>
files<font color="#990000">:</font> <font color="#990000">[</font>
<font color="#FF0000">{</font>
expand<font color="#990000">:</font> <b><font color="#0000FF">true</font></b><font color="#990000">,</font>
cwd<font color="#990000">:</font> <font color="#FF0000">'app'</font><font color="#990000">,</font>
src<font color="#990000">:</font> <font color="#990000">[</font>
<font color="#FF0000">'images/*.{png,gif,jpg,svg}'</font><font color="#990000">,</font>
<font color="#FF0000">'*.html'</font>
<font color="#990000">],</font>
dest<font color="#990000">:</font> <font color="#FF0000">'dist/app'</font>
<font color="#FF0000">}</font>
<font color="#990000">]</font>
<font color="#FF0000">}</font>
<font color="#FF0000">}</font>
</tt></pre></blockquote>
<p>Here I use a different configuration for the files. The <code>expand</code> option is what
is important. I tells grunt to copy the files preserving the structure.</p>
<p>OK, now all the files have been moved into their proper place and all that is
left is to checksum them and rename all the references.</p>
<h2>Filerev, checksumming</h2>
<p><code>filerev</code> is my task of choice for adding the checksum of a file to its name. I
use MD5 to checksum all assets, javascript, css and images with this
configuration.</p>
<blockquote><pre><tt><i><font color="#9A1900">// Filerev</font></i>
filerev<font color="#990000">:</font> <font color="#FF0000">{</font>
options<font color="#990000">:</font> <font color="#FF0000">{</font>
encoding<font color="#990000">:</font> <font color="#FF0000">'utf8'</font><font color="#990000">,</font>
algorithm<font color="#990000">:</font> <font color="#FF0000">'md5'</font><font color="#990000">,</font>
length<font color="#990000">:</font> <font color="#993399">20</font>
<font color="#FF0000">}</font><font color="#990000">,</font>
release<font color="#990000">:</font> <font color="#FF0000">{</font>
<i><font color="#9A1900">// filerev:release hashes(md5) all assets (images, js and css )</font></i>
<i><font color="#9A1900">// in dist directory</font></i>
files<font color="#990000">:</font> <font color="#990000">[</font><font color="#FF0000">{</font>
src<font color="#990000">:</font> <font color="#990000">[</font>
<font color="#FF0000">'dist/app/images/*.{png,gif,jpg,svg}'</font><font color="#990000">,</font>
<font color="#FF0000">'dist/app/scripts/*.js'</font><font color="#990000">,</font>
<font color="#FF0000">'dist/app/styles/*.css'</font><font color="#990000">,</font>
<font color="#990000">]</font>
<font color="#FF0000">}</font><font color="#990000">]</font>
<font color="#FF0000">}</font>
<font color="#FF0000">}</font>
</tt></pre></blockquote>
<h2>Usemin</h2>
<p>The final task is to change all the references in the HTML and CSS files to use
the checksummed filenames and to change the <code>script</code> tags to reference the
minified file. <code>usemin</code> is the task for this job.</p>
<blockquote><pre><tt><i><font color="#9A1900">// Usemin</font></i>
<i><font color="#9A1900">// Replaces all assets with their revved version in html and css files.</font></i>
<i><font color="#9A1900">// options.assetDirs contains the directories for finding the assets</font></i>
<i><font color="#9A1900">// according to their relative paths</font></i>
usemin<font color="#990000">:</font> <font color="#FF0000">{</font>
html<font color="#990000">:</font> <font color="#990000">[</font><font color="#FF0000">'dist/app/*.html'</font><font color="#990000">],</font>
css<font color="#990000">:</font> <font color="#990000">[</font><font color="#FF0000">'dist/app/styles/*.css'</font><font color="#990000">],</font>
options<font color="#990000">:</font> <font color="#FF0000">{</font>
assetsDirs<font color="#990000">:</font> <font color="#990000">[</font><font color="#FF0000">'dist/app'</font><font color="#990000">,</font> <font color="#FF0000">'dist/app/styles'</font><font color="#990000">]</font>
<font color="#FF0000">}</font>
<font color="#FF0000">}</font>
</tt></pre></blockquote>
<p>The only difficult thing about this is that <code>usemin</code> uses the paths from the
files it parses when it searches for assets to replace references to. This
means that <code>options.assetsDirs</code> must designate the directories where the
parsed files are located. In my case the CSS files are in <code>dist/app/styles</code> and
the HTML files are in <code>dist/app</code>. Hoohaah! Only one more thing before were
done. Calling all the tasks in order.</p>
<h2>Release</h2>
<p>I register the <code>release</code> task and tell it to invoke all the other files in
the correct order.</p>
<blockquote><pre><tt><i><font color="#9A1900">// Invoked with grunt release, creates a release structure</font></i>
grunt<font color="#990000">.</font><b><font color="#000000">registerTask</font></b><font color="#990000">(</font><font color="#FF0000">'release'</font><font color="#990000">,</font> <font color="#FF0000">'Creates a release in /dist'</font><font color="#990000">,</font> <font color="#990000">[</font>
<font color="#FF0000">'clean'</font><font color="#990000">,</font>
<font color="#FF0000">'jshint'</font><font color="#990000">,</font>
<font color="#FF0000">'less:release'</font><font color="#990000">,</font>
<font color="#FF0000">'useminPrepare'</font><font color="#990000">,</font>
<font color="#FF0000">'concat'</font><font color="#990000">,</font>
<font color="#FF0000">'uglify'</font><font color="#990000">,</font>
<font color="#FF0000">'copy'</font><font color="#990000">,</font>
<font color="#FF0000">'filerev'</font><font color="#990000">,</font>
<font color="#FF0000">'usemin'</font>
<font color="#990000">]);</font>
</tt></pre></blockquote>
<h2>Example Code</h2>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiN47OQVx0rU96W7KKRH4UTrlwwOnh41KlN3s7Foihk4KfyjNQs0iUJq7N-YVLaVK8IjgHZWbF2cng2lOs2OKM5qBzTn_CY90ORkO_eblFXXMftpK-_56DHFpb0r86QxpAhKUkp/s1600/grunt.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiN47OQVx0rU96W7KKRH4UTrlwwOnh41KlN3s7Foihk4KfyjNQs0iUJq7N-YVLaVK8IjgHZWbF2cng2lOs2OKM5qBzTn_CY90ORkO_eblFXXMftpK-_56DHFpb0r86QxpAhKUkp/s320/grunt.png" /></a></div>
<p>This example comes from a workshop I give. If you are interested in one send me
a note. If you would like to give one yourself you are welcome to use
<a href="https://github.com/andersjanmyr/grunt-lab">my example code</a>. I also give a
<a href="http://andersjanmyr.github.io/grunt-presentation">Grunt presentation</a></p>
<p>That's all folks!</p>Anders Janmyrhttp://www.blogger.com/profile/16081537018884349622noreply@blogger.com7tag:blogger.com,1999:blog-34049130.post-87526205997616506272014-01-19T08:43:00.002+01:002014-01-19T08:43:57.940+01:00Designing Programs, Caballo Blanco Style<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgH3ogkG4fZb_9yR_7mFlZnCHqXzVQCJEB5Px_RnbEPplgtJs0DWL4aUGgVHODjL0eD9AV5NljDKJmOhCYrP-SvwYIINXJQfAIz8k9ZKWlJxKz0i_5FvLjanaxitkoDmo_kSC61/s1600/caballo-blanco+(1).jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgH3ogkG4fZb_9yR_7mFlZnCHqXzVQCJEB5Px_RnbEPplgtJs0DWL4aUGgVHODjL0eD9AV5NljDKJmOhCYrP-SvwYIINXJQfAIz8k9ZKWlJxKz0i_5FvLjanaxitkoDmo_kSC61/s320/caballo-blanco+(1).jpg" /></a></div>
<p>I enjoy picking up new ideas on programming from different sources. It is
cool how the mind can take an idea from one domain and naturally fit it into a
totally different one. In this case I have picked up some programming tips from
Caballo Blanco, the lone ultrarunner from the book,
<a href="http://www.amazon.com/Born-Run-Hidden-Superathletes-Greatest/dp/0307279189?tag=thtasta-20">Born to Run</a></p>
<p>It is a great book even if you are not into running at all and it also
contains tips about programming. (Of course this depends on having a warped
mind, like I do :)</p>
<p>The text below is from the book but I have altered the wording to put it into a
programming context.</p>
<blockquote>
<p>Don't fight the code, take what it gives you. If you have a choice between one
function or two, make three!</p>
<p>Caballo has spent so many years solving problems so he even has nicknames for
them. Some are <em>ayudantes</em>, reliable problems which behave as you expect,
problems that look and <em>are</em> straightforward to solve. Others are <em>tricksters</em>
which look like ayudantes but will become difficult when you start digging into
them. Some are <em>chingón sitos</em>, little bastards, just dying to trip you up.</p>
<p>Think, easy, light, smooth, and fast!</p>
<p>You start with easy because if that is all you get, that's not so bad!
Then, work on light. Make it effortless like you don't give a shit how big the
problem is or how long it is going to take. When you have practiced this so
long that you have forget you are practicing, you work on getting it smooooth!
You don't have to worry about the last one. If you get those three and you'll
be fast!</p>
</blockquote>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-TDQ0Ts2fhOAbprR5sRBOcr_xXCvKCoLuAQ_ozd54yA_1uFWsrhKQlSgypUpicXqTrKjVkfdVCIiwhFB4FCUZn7YWUdTa49X_MayTwi2yKgmjshSCJl5bZYK2qjkxaKGirJgf/s1600/caballo-blanco.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-TDQ0Ts2fhOAbprR5sRBOcr_xXCvKCoLuAQ_ozd54yA_1uFWsrhKQlSgypUpicXqTrKjVkfdVCIiwhFB4FCUZn7YWUdTa49X_MayTwi2yKgmjshSCJl5bZYK2qjkxaKGirJgf/s320/caballo-blanco.jpg" /></a></div>
<p>I like this text because it makes it clear that what is important is to focus
on the doing and not on being done. We don't need to think about being fast
because if we fully focus on the problem at hand, we will automatically be as
fast as we can!</p>
Anders Janmyrhttp://www.blogger.com/profile/16081537018884349622noreply@blogger.com0tag:blogger.com,1999:blog-34049130.post-58223000967836609352013-12-25T10:06:00.001+01:002013-12-26T08:57:45.938+01:00A Consultant's Wishlist
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhyVE1C4njMdGsFJ_i4EUDp39n-HSiJDgS6SgemfJGREN61i33T5sptVX-mP6xEtVH38wgTE1OJP8_dxKG6nD6CAmAjT64fCSUnlXCBSYG7cWcKAH4ZZMWtxUUmv2hSyJOoxX3o/s1600/wishlist_santa.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhyVE1C4njMdGsFJ_i4EUDp39n-HSiJDgS6SgemfJGREN61i33T5sptVX-mP6xEtVH38wgTE1OJP8_dxKG6nD6CAmAjT64fCSUnlXCBSYG7cWcKAH4ZZMWtxUUmv2hSyJOoxX3o/s320/wishlist_santa.jpg" /></a></div>
<p>As a consultant I am called in to solve specific problems and I would like to
start working on the problems at hand as soon as I can. But, very often, there
are other problems that I have to overcome before I am able to spend my time on
the problems I am being paid to solve. Here is my wishlist.</p>
<h2>A Reliable Internet Connection</h2>
<p>Most people in Sweden have a really good Internet connections, but it is
surprising how many big companies don't. Even more surprising is that it is not
seen as a major problem. A consultant usually costs at least 100 euro an hour,
having us idle because of a slow network is bad business practice.</p>
<p>Some companies even take it further. If the connection is not flaky enough they
make sure there is an even more flaky proxy that will make sure that every task
performed will take special work since many of the tools we use for daily web
development are not easily configured to work with a proxy.</p>
<p>Please give me fast, reliable networking without obstructions.</p>
<h2>Personal Computer</h2>
<p>I want to work with my personal computer. I have configured it to make me as
effective as possible and I don't want to be forced to work on another computer
just because that is corporate policy. Having my personal computer also allows
me to work from wherever I am, at home, on the train, etc. Sure, I can
configure the computer that I am given to work the way I want it to work,
but that is time spent on a different problem than I was hired to solve.</p>
<h2>Software as a Service</h2>
<p>I also find it frustrating to have to use legacy tools. Many of them are not
working well and I would like to do without them. We are living in the era of
Software as a Service and a lot of the best tools are provided as services and
as a consultant I already have accounts for them and it is only a click a way
to allow me to become part of the company team.</p>
<h3>Email</h3>
<p>I have an email address. A remarkable thing about an email address is that it works no matter where I am.
I don't want to use corporate email and be part of various groups that I care nothing about.
I want to use my own email that I can access from anywhere through a good interface.</p>
<h3>Github</h3>
<p>If a company is using Github that means that I can be setup and working on the
source code in a matter of minutes.</p>
<p>Apart from being easy to setup and familiar to me, Github has a number of
advantages over other solutions.</p>
<ul>
<li>Built-in markdown parsing allowing documentation with simple navigation to
other documents, issues, code, pull-requests and commits.</li>
<li>Issue tracking seamlessly integrated with version control allowing for easy
tracking of issues through the code.</li>
<li>Visualization of branches allows developers and other interested parties to
see what is being worked on at any time.</li>
<li>Pull-requests is a simple way for developers to do code-reviews and to
communicate publicly about code.</li>
</ul>
<h3>Trello</h3>
<p>Most teams I work with often work (or try to work) with an agile process. This
often means that we use a Kanban board of some kind. I have tried a number of
them and Trello is the one I like the most. It is lightweight, provides (near)
real time updates to the board, and it has clients for Web, Android and IOS.
This makes it easy to see what is being worked on and easy to update or add
new tasks even if I am not at the computer.</p>
<h3>Travis CI</h3>
<p>Travis is continuous integration as a service. It is ridiculously easy to set
up and it integrates really well with Github. I know that it is easy to setup
up a Jenkins server to do this but what is the point when Travis is already
setup and running.</p>
<h3>Campfire or Skype</h3>
<p>Campfire and Skype are awesome tools for collaborating teams who want to keep in sync across time
and space.</p>
<h2>Platform as a Service</h2>
<p>I also find it much better if the product being worked on is hosted by a
service provided that can be easily accessed from anywhere. Depending on what
level of control you need there are a number of options. I like Heroku and
Nodejitsu for hosting Ruby and Node and it is often all I really need.
If I need more control it is easy to switch to Amazon or another IaaS providers
if the need comes up.</p>
<h2>Freedom</h2>
<p>I want the best for my clients. To do this I want to work with really good
tools, from anywhere. I want to work in an environment that keeps me happy and
productive. Merry Christmas!</p>
Anders Janmyrhttp://www.blogger.com/profile/16081537018884349622noreply@blogger.com0tag:blogger.com,1999:blog-34049130.post-64710662783534861652013-11-06T13:45:00.000+01:002013-11-07T08:37:50.115+01:00References from Habits of a Responsible Programmer
<p>A list of references from my talk, <a href="http://oredev.org/2013/wed-fri-conference/habits-of-a-responsible-programmer">Habits of a Responsible Programmer</a> at Øredev. First out is <a href="http://anders.janmyr.com/2013/04/a-responsible-programmer.html">the blog post that inspired the talk</a></p>
<h3>Habits</h3>
<p>Some books about habits and how our brain works in good and not so good ways.</p>
<ul>
<li><a href="http://www.amazon.com/Thinking-Fast-Slow-Daniel-Kahneman/dp/0374275637?tag=thtasta-20">Thinking Fast and Slow</a> by Daniel Kahnemann</li>
<li><a href="http://www.amazon.com/The-Power-Habit-What-Business/dp/1400069289?tag=thtasta-20">The Power of Habit</a> by Charles Duhigg</li>
<li><a href="http://www.amazon.com/Blink-The-Power-Thinking-Without/dp/0316010669?tag=thtasta-20">Blink</a> by Malcolm Gladwell</li>
</ul>
<h3>Typing</h3>
<p>Steve Yegges blog post that made me realize that it is time to learn to touch
type along with some tools that can make it fun to learn.</p>
<ul>
<li><a href="http://steve-yegge.blogspot.se/2008/09/programmings-dirtiest-little-secret.html">Programming's Dirtiest Little Secret</a> by Steve Yegge</li>
<li><a href="http://atypetrainer4mac.en.softonic.com/mac">aTypeTrainer</a></li>
<li><a href="http://play.typeracer.com/">Type Racer</a></li>
</ul>
<h3>Clear Code</h3>
<p>It is worth learning Smalltalk, just to understand the book, Smalltalk Best
Practice Patterns.</p>
<ul>
<li><a href="http://www.amazon.com/gp/product/013476904X?tag=thtasta-20">Smalltalk Best Practice Patterns</a> by Kent Beck</li>
<li><a href="http://www.amazon.com/gp/product/0132350882?tag=thtasta-20">Clean Code</a> by
Bob Martin</li>
</ul>
<h3>Programming Techniques</h3>
<p>SICP is a classic and it deserves to be read but Concepts, Techniques, and
Models of Computer Programming is just as good if not better.</p>
<ul>
<li><a href="http://www.amazon.com/gp/product/0262220695?tag=thtasta-20">Concepts, Techniques, and Models of Computer Programming</a> by Peter van Roy and Seif Haridi</li>
<li><a href="http://www.amazon.com/gp/product/0262011530?tag=thtasta-20">Structure and Interpretation of Computer Programs</a> by Harold Abelson and Jay Sussman</li>
<li><a href="http://www.amazon.com/gp/product/0262560992?tag=thtasta-20">The Little Schemer</a> by Daniel P. Friedman and Matthias Felleisen</li>
<li><a href="http://www.amazon.com/gp/product/026256100X?tag=thtasta-20">The Seasoned Schemer</a> by Daniel P. Friedman and Matthias Felleisen</li>
<li><a href="http://anders.janmyr.com/2013/04/javascript-conditionals.html">Javascript Conditionals</a> by me</li>
</ul>
<h3>Read Code</h3>
<p>Source code to a number of projects with beautiful code.</p>
<ul>
<li><a href="https://github.com/cloudfoundry">CloudFoundry</a> - beautiful Ruby code, OS
Platform as a Service.</li>
<li><a href="https://github.com/rvagg/node-levelup">Levelup</a> - Node wrapper for the
LevelDB key-value store.</li>
</ul>
<h3>Explicit</h3>
<p>A good short article by Marin Fowler on the tradeoffs involved with writing
explicit or implicit code.</p>
<ul>
<li><a href="http://martinfowler.com/ieeeSoftware/explicit.pdf">To Be Explicit (pdf)</a> by Martin Fowler</li>
</ul>
<h3>Refactoring</h3>
<p>The book by Fowler is a timeless reference book and it is required reading for
anyone serious about programming. Kerievsky's book gives more in depth examples
and finally Reg's blog post discusses reasons for not refactoring just to
please your own ego.</p>
<ul>
<li><a href="http://www.amazon.com/gp/product/0201485672?tag=thtasta-20">Refactoring: Improving the Design of Existing Code</a> by Martin Fowler</li>
<li><a href="http://www.amazon.com/Refactoring-Patterns-Joshua-Kerievsky/dp/0321213351?tag=thtasta-20">Refactoring to Patterns</a> by Joshua Kerievsky</li>
<li><a href="http://weblog.raganwald.com/2008/05/narcissism-of-small-code-differences.html">The Narcissism of Small Code Differences</a> by Reginald Braithwaite</li>
</ul>
<h3>Simple vs. Easy</h3>
<p>Great talk!</p>
<ul>
<li><a href="http://www.infoq.com/presentations/Simple-Made-Easy">Simple Made Easy</a> by
Rich Hickey</li>
</ul>
<h3>Testing</h3>
<p>I like Sandi Metz way of explaining why testing is important.</p>
<ul>
<li><a href="http://www.youtube.com/watch?v=URSWYvyc42M">Magic Tricks of Testing</a> by Sandi Metz</li>
<li><a href="http://www.amazon.com/Sandi-Metz/e/B0097WWH62?tag=thtasta-20">Practical Object Oriented Desgin in Ruby</a> by Sandi Metz</li>
<li><a href="http://www.objectmentor.com/resources/publishedArticles.html">The Craftsman Articles</a> by Bob Martin (click on the Craftsman link).</li>
<li><a href="http://teacher.nsrl.rochester.edu/phy_labs/appendixe/appendixe.html">The Scientific Method</a></li>
</ul>
<h3>Documentation</h3>
<p>Documentation matters, but keep it simple.</p>
<ul>
<li><a href="http://tom.preston-werner.com/2010/08/23/readme-driven-development.html">Readme Driven Development</a></li>
<li><a href="http://www.infoq.com/articles/ddd-contextmapping">DDD Context Map</a></li>
</ul>
<h3>Git</h3>
<p>The original gitcasts.com is no longer available, but the videos are uploaded
to You Tube.</p>
<ul>
<li><a href="http://anders.janmyr.com/search?q=git">Articles on Git</a> from my blog.</li>
<li><a href="http://www.youtube.com/playlist?list=PLttwD7NyH3omQLyVtan0CFOX_UWItX_yG">Git Casts on You Tube</a></li>
</ul>
<h3>Scripting</h3>
<ul>
<li><a href="http://www.amazon.com/Unix-Power-Tools-Third-Edition/dp/0596003307?tag=thtasta-20">Unix Power Tools</a> by Tim O'Reilly et al.</li>
<li><a href="http://www.amazon.com/Bash-Cookbook-Solutions-Examples-Cookbooks/dp/0596526784?tag=thtasta-20">Bash Cookbook</a> by Carl Albing et al.</li>
<li><a href="http://www.amazon.com/Build-Awesome-Command-Line-Applications-Ruby/dp/1934356913?tag=thtasta-20">Build Awesome Command-Line Applications in Ruby</a> by David Copeland</li>
</ul>
<h3>Generating Code</h3>
<p>Tools for generation code. Here documents are the simplest possible way, but if
you want to generate multiple files it is better to use Thor or Yo.</p>
<ul>
<li><a href="http://www.tldp.org/LDP/abs/html/here-docs.html">Here documents</a> (Bash)</li>
<li><a href="http://whatisthor.com/">Thor</a> (Ruby)</li>
<li><a href="https://github.com/yeoman/yo">Yo</a> (Javascript)</li>
</ul>
<h3>Environments</h3>
<p>The ultimate book on continuous delivery, a must read for anyone interested in
automation. And, that should be everyone!</p>
<ul>
<li><a href="http://www.amazon.com/Continuous-Delivery-Deployment-Automation-Addison-Wesley/dp/0321601912?tag=thtasta-20">Continuous Delivery</a> by Jez Humble and David Farley</li>
</ul>
<h3>Projects</h3>
<p>Wonderful book about different people stereotypes.</p>
<ul>
<li><a href="http://www.amazon.com/Adrenaline-Junkies-Template-Zombies-Understanding/dp/0932633676?tag=thtasta-20">Adrenaline Junkies and Template Zombies</a> by Tom Demarco et al</li>
</ul>
<h3>Estimation</h3>
<p>Two articles on estimation by Martin Fowler and Dan North.</p>
<ul>
<li><a href="http://dannorth.net/2013/08/08/blink-estimation/">Blink Estimation</a> by Dan North</li>
<li><a href="http://martinfowler.com/bliki/PurposeOfEstimation.html">The Purpose of Estimation</a> by Martin Fowler</li>
<li><a href="http://anders.janmyr.com/2010/08/no-deadlines.html">No Deadlines</a> by me</li>
</ul>
<h3>Life, the Universe, and Everything</h3>
<p>Books about happiness, the mind, and other things.</p>
<ul>
<li><a href="http://www.amazon.com/gp/product/0060920432?&amp;tag=thtasta-20">Flow</a> by Mihaly Csikszentmihalyi</li>
<li><a href="http://www.amazon.com/Your-Brain-Work-Strategies-Distraction/dp/0061771295?tag=thtasta-20">Your Brain at Work</a> by David Rock</li>
<li><a href="http://www.amazon.com/Habits-Highly-Effective-People-Powerful/dp/1455892823">7 Habits of Highly Effective People</a> by Stephen Covey</li>
<li><a href="http://www.amazon.com/The-Silva-Mind-Control-Method/dp/0671739891?tag=thtasta-20">The Silva Mind Control Method</a> by José Silva</li>
<li><a href="http://www.amazon.com/Authentic-Happiness-ebook/dp/B000FBJFWI?tag=thtasta-20">Authentic Happiness</a> by Martin Seligman</li>
<li><a href="http://www.amazon.com/Opening-Door-Your-Heart-Happiness/dp/1743107161">Opening the Door of your Heart</a> by Ajahn Brahm</li>
</ul>Anders Janmyrhttp://www.blogger.com/profile/16081537018884349622noreply@blogger.com0tag:blogger.com,1999:blog-34049130.post-34843018428824719442013-10-17T16:40:00.000+02:002013-10-17T16:40:47.003+02:00Tunneling to localhost via SSH<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGXh6bDIDGEL7pm7iKBaqfqLnIuzLD0ipeJ9LS1uqlZ3vLEec1idC8s1WOjKp1pmrIlkF1jeLCRNayVHtVTeCsqWrVoe1UQ2llkynQyZ3Q2hci7fHyVCNa5l41BQE9qO6ECexm/s1600/tunnel.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGXh6bDIDGEL7pm7iKBaqfqLnIuzLD0ipeJ9LS1uqlZ3vLEec1idC8s1WOjKp1pmrIlkF1jeLCRNayVHtVTeCsqWrVoe1UQ2llkynQyZ3Q2hci7fHyVCNa5l41BQE9qO6ECexm/s320/tunnel.jpg" /></a></div>
<p>Sometimes when working with a new web site I have customers who want to see the
site while it is still in development. One way of doing this
is to have alternative demo servers where all we do is just serve up our work
in progress. This works fine most of the time, but sometimes I just want to be
serve up my local machine.</p>
<h2>Localtunnel</h2>
<p>One easy way to do this is to use <a href="http://progrium.com/localtunnel/">localtunnel</a>.
<code>localtunnel</code> is a Ruby gem that is meant for exactly this purpose. Here's how:</p>
<blockquote><pre><tt>$ gem install localtunnel
$ localtunnel --key <font color="#990000">~/.</font>ssh/id_rsa<font color="#990000">.</font>pub <font color="#993399">80</font>
This localtunnel service is brought to you by Twilio<font color="#990000">.</font>
Port <font color="#993399">80</font> is now publicly accessible from http<font color="#990000">://</font>4xiw<font color="#990000">.</font>localtunnel<font color="#990000">.</font>com <font color="#990000">...</font>
</tt></pre></blockquote>
<p>It is now possible to access <code>localhost:80</code> via the URL
<code>http://4xiw.localtunnel.com</code>. Simple as pie!</p>
<p>But what if you don't like pie? Or Ruby, for that matter. Or, what if you don't
like to serve your secret data through another company (Twilio in this case who
graciously gives away the hosting for free).</p>
<p>Well you are in luck, it is easy to set up your own tunnel via SSH, provided
you have access to a server that is accessible from the Internet. And,
everybody has access to such a server via Amazon EC2 or similar service. Make
sure the server is accessible on all high ports. On Amazon this is done by
opening all incoming ports above 1024 in a security group.</p>
<h2>Setting up a tunnel via SSH</h2>
<p>In SSH lingo a tunnel from an external server to my local server is called a
reverse proxy. Here is how to set one up. First you need to configure the
remote ssh daemon to allow setting up remote interfaces. A remote interface is
one that can be accessed from a server other than localhost, which is what we
want.</p>
<p>Here is how to do it, the server called <code>host.amazonaws.com</code> is my AWS server.</p>
<blockquote><pre><tt> <i><font color="#9A1900"># Login to the remote server</font></i>
$ ssh -i <font color="#990000">~/.</font>ssh/id_rsa ec2-user@host<font color="#990000">.</font>amazonaws<font color="#990000">.</font>com
<i><font color="#9A1900"># Edit the sshd configuration</font></i>
$ sudo vi /etc/ssh/sshd_config
<i><font color="#9A1900"># Find the line #GatewayPorts no</font></i>
<i><font color="#9A1900"># Change it to GatewayPorts yes</font></i>
<i><font color="#9A1900"># Save and exit</font></i>
<i><font color="#9A1900"># Restart the daemon</font></i>
$ sudo /etc/init<font color="#990000">.</font>d/sshd restart
Stopping sshd<font color="#990000">:</font> <font color="#990000">[</font> OK <font color="#990000">]</font>
Starting sshd<font color="#990000">:</font> <font color="#990000">[</font> OK <font color="#990000">]</font>
<i><font color="#9A1900"># Exit the shell and return to your local machine</font></i>
$ <b><font color="#0000FF">exit</font></b>
</tt></pre></blockquote>
<p>Now you are good to go. We assume you have a server running on port 3000 to
display to the world.</p>
<blockquote><pre><tt> $ ssh -i <font color="#990000">~/.</font>ssh/id_rsa -N -R <font color="#990000">*:</font><font color="#993399">0</font><font color="#990000">:</font>localhost<font color="#990000">:</font><font color="#993399">3000</font> ec2-user@host<font color="#990000">.</font>amazonaws<font color="#990000">.</font>com
Allocated port <font color="#993399">34070</font> <b><font color="#0000FF">for</font></b> remote forward to localhost<font color="#990000">:</font><font color="#993399">3000</font>
</tt></pre></blockquote>
<p>Now you can surf to <code>http://host.amazonaws.com:34070</code></p>
<p>And it will access your local machine. :)</p>
<p>When you stop the command (Ctrl-C) the tunneling will stop.</p>
<h3>Command explanation</h3>
<pre><code>-i identity file (private key)
-N Do not execute a remote command, just setup the port forwarding
-R * All interfaces should be forwarded
0 Open forwarding on any available port (34070 in the example)
localhost Forward to localhost
3000 The localport to forward too.
ec2-user The user on the remote server
host.aws.com The remote server
</code></pre>
<p>If you want to simplify it for yourself add the following script to a bin
catalog.</p>
<blockquote><pre><tt> <i><font color="#9A1900">## Script tunnel</font></i>
<i><font color="#9A1900">#!/bin/bash</font></i>
<b><font color="#0000FF">set</font></b> -o errexit
<i><font color="#9A1900"># default to port 3000</font></i>
<font color="#009900">port</font><font color="#990000">=</font><font color="#009900">${1:-3000}</font>
ssh -i <font color="#990000">~/.</font>ssh/id_rsa -N -R <font color="#990000">\*:</font><font color="#993399">0</font><font color="#990000">:</font>localhost<font color="#990000">:</font><font color="#009900">$port</font> ec2-user@host<font color="#990000">.</font>amazonaws<font color="#990000">.</font>com
</tt></pre></blockquote>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiMrvfVXh2hId0PgNP6Dn-MV68DsSN3QZpVm6aHkHujsFPrnjeCiT11lJ1fDe64RWRfLQREpTmQohMqfbKGduYDTvTJ5K66WOiyNEWa0mCUtBpXl8aVCJjAvFKNRjzhIPNdNrRp/s1600/jail.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiMrvfVXh2hId0PgNP6Dn-MV68DsSN3QZpVm6aHkHujsFPrnjeCiT11lJ1fDe64RWRfLQREpTmQohMqfbKGduYDTvTJ5K66WOiyNEWa0mCUtBpXl8aVCJjAvFKNRjzhIPNdNrRp/s320/jail.jpg" /></a></div>
<p>Now all you have to do to enable remote access is <code>tunnel 80</code>, or whatever port
you want to display.</p>
<h2>Escaping the proxy Jail</h2>
<p>The story could have ended here, but some people, trapped behind corporate
firewalls, may not be allowed to use <code>ssh</code>. The traffic is blocked by a
corporate proxy server. Well, there is a happy ending for you too and it is
fittingly called <a href="http://www.agroman.net/corkscrew/">corkscrew</a>. It allows you
to screw yourself out of the corporate jail and into the world.</p>
<p>Here is how you do it on Ubuntu, on OS X use <code>brew</code> instead.</p>
<br/>
<blockquote><pre><tt> <i><font color="#9A1900"># Install corkscrew</font></i>
$ sudo apt-get install corkscrew
<i><font color="#9A1900"># Edit your ~/.ssh/config, add</font></i>
Host <font color="#990000">*</font>
ProxyCommand corkscrew http-proxy<font color="#990000">.</font>example<font color="#990000">.</font>com <font color="#993399">8080</font> <font color="#990000">%</font>h <font color="#990000">%</font>p
<i><font color="#9A1900"># If you need to authenticate to get through the proxy the line should read</font></i>
Host <font color="#990000">*</font>
ProxyCommand corkscrew http-proxy<font color="#990000">.</font>example<font color="#990000">.</font>com <font color="#993399">8080</font> <font color="#990000">%</font>h <font color="#990000">%</font>p <font color="#990000">~/.</font>ssh/proxyauth
<i><font color="#9A1900"># And you need to add username:password to ~/.ssh/proxyauth</font></i>
$ echo <font color="#FF0000">"proxyusername:proxypassword"</font> <font color="#990000">></font> <font color="#990000">~/.</font>ssh/proxyauth
</tt></pre></blockquote>
<p><code>%h</code> and <code>%p</code> is filled in by <code>ssh</code> with the host and port of your destination.</p>
<blockquote><pre><i>Freedom's just another word for nothing left to lose,
Nothin' don't mean nothin', honey, if it ain't free.
Yeah, feeling good was easy, Lord, when he sang the blues,
You know feeling good was good enough for me,
Good enough for me and my Bobby McGee.
</i></pre></blockquote>
Anders Janmyrhttp://www.blogger.com/profile/16081537018884349622noreply@blogger.com0tag:blogger.com,1999:blog-34049130.post-64074457198819358482013-06-20T12:25:00.000+02:002014-03-03T09:15:33.267+01:00Solving the Expression Problem in Javascript<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhhFHPjjpQkxV2BagfFLtRb_6jhAu_amjNuF_Klw3qks6hmFaipKsZNhW3pfgJ6F5BMkyaa78S9yhT6YrabZba05ZOIXxMNqf0iCZg_scQTqIdntpD3sF8zGQCcBbDJQ_9Vc4sH/s1600/madonna_express_yourself.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhhFHPjjpQkxV2BagfFLtRb_6jhAu_amjNuF_Klw3qks6hmFaipKsZNhW3pfgJ6F5BMkyaa78S9yhT6YrabZba05ZOIXxMNqf0iCZg_scQTqIdntpD3sF8zGQCcBbDJQ_9Vc4sH/s320/madonna_express_yourself.jpg" /></a></div>
<p>
I just watched a great presentation by Daniel Spiewak called
<a href="https://vimeo.com/user18356272/review/66548717/3531875329">Living in a Post-Functional World</a>.
I watched it mainly because I heard it was a great presentation on how to deal
with modules, which it was. A concept which is just as important in Javascript as
it is in Scala.</p>
<p>But at the end of the presentation Daniel talks about the
<a href="http://www.daimi.au.dk/%7Emadst/tool/papers/expression.txt">Expression Problem</a>
as defined by <a href="http://homepages.inf.ed.ac.uk/wadler/">Philip Wadler</a>.</p>
<p>Here it is as summarized by Daniel Spiewak:</p>
<h2>The Expression Problem</h2>
<ul>
<li>Define a datatype by cases</li>
<li>Add new cases to the datatype</li>
<li>Add new functions over the datatype</li>
<li>Don't recompile</li>
<li><strong>Good Luck!</strong></li>
</ul>
<h2>Functional Style</h2>
<p>If we try to solve the problem in a functional style, we get something like
this (also from Daniel's presentation).</p>
<blockquote><pre><tt><b><font color="#0000FF">sealed</font></b> <b><font color="#0000FF">trait</font></b> <font color="#008080">Expr</font>
<b><font color="#0000FF">case</font></b> <b><font color="#0000FF">class</font></b> <font color="#008080">Add</font><font color="#990000">(</font>e1<b><font color="#0000FF">:</font></b> Expr<font color="#990000">,</font> e2<b><font color="#0000FF">:</font></b> Expr<font color="#990000">)</font> <b><font color="#0000FF">extends</font></b> Expr
<b><font color="#0000FF">case</font></b> <b><font color="#0000FF">class</font></b> <font color="#008080">Sub</font><font color="#990000">(</font>e1<b><font color="#0000FF">:</font></b> Expr<font color="#990000">,</font> e2<b><font color="#0000FF">:</font></b> Expr<font color="#990000">)</font> <b><font color="#0000FF">extends</font></b> Expr
<b><font color="#0000FF">case</font></b> <b><font color="#0000FF">class</font></b> <font color="#008080">Num</font><font color="#990000">(</font>n<b><font color="#0000FF">:</font></b> <font color="#009900">Int</font><font color="#990000">)</font> <b><font color="#0000FF">extends</font></b> Expr
<b><font color="#0000FF">def</font></b> <b><font color="#000000">value</font></b><font color="#990000">(</font>e<b><font color="#0000FF">:</font></b> Expr<font color="#990000">)</font><b><font color="#0000FF">:</font></b> <font color="#009900">Int</font> <b><font color="#0000FF">=</font></b> e <b><font color="#0000FF">match</font></b> <font color="#FF0000">{</font>
<b><font color="#0000FF">case</font></b> <b><font color="#000000">Add</font></b><font color="#990000">(</font>e1<font color="#990000">,</font> e2<font color="#990000">)</font> <b><font color="#0000FF">=></font></b>
<b><font color="#000000">value</font></b><font color="#990000">(</font>e1<font color="#990000">)</font> <font color="#990000">+</font> <b><font color="#000000">value</font></b><font color="#990000">(</font>e2<font color="#990000">)</font>
<b><font color="#0000FF">case</font></b> <b><font color="#000000">Sub</font></b><font color="#990000">(</font>e1<font color="#990000">,</font> e2<font color="#990000">)</font> <b><font color="#0000FF">=></font></b>
<b><font color="#000000">value</font></b><font color="#990000">(</font>e1<font color="#990000">)</font> <font color="#990000">-</font> <b><font color="#000000">value</font></b><font color="#990000">(</font>e2<font color="#990000">)</font>
<b><font color="#0000FF">case</font></b> <b><font color="#000000">Num</font></b><font color="#990000">(</font>n<font color="#990000">)</font> <b><font color="#0000FF">=></font></b> n
<font color="#FF0000">}</font>
</tt></pre></blockquote>
<p>The functional style uses <em>pattern matching</em>. We see that it is easy to add new
functions, such as a <code>toString</code> that returns a string representation of the
expression without changing any code. If we add a new class, such as <code>Mul</code>, we
have to change all the existing functions.</p>
<p>Here are the main points of this solution:</p>
<ul>
<li>Dumb cases</li>
<li>Every function enumerates full algebra</li>
<li>Very easy to add new functions</li>
<li>Very difficult to add new cases</li>
</ul>
<p>We get an <em>open set of functions and a closed set of cases!</em></p>
<h2>Object-Oriented Style</h2>
<p>If we try to solve the problem in an object-oriented style, we get something
like this (again from Daniel's presentation).</p>
<blockquote><pre><tt><b><font color="#0000FF">sealed</font></b> <b><font color="#0000FF">trait</font></b> <font color="#008080">Expr</font> <font color="#FF0000">{</font>
<b><font color="#0000FF">def</font></b> value<b><font color="#0000FF">:</font></b> <font color="#009900">Int</font>
<font color="#FF0000">}</font>
<b><font color="#0000FF">case</font></b> <b><font color="#0000FF">class</font></b> <font color="#008080">Add</font><font color="#990000">(</font>e1<b><font color="#0000FF">:</font></b> Expr<font color="#990000">,</font> e2<b><font color="#0000FF">:</font></b> Expr<font color="#990000">)</font> <b><font color="#0000FF">extends</font></b> Expr
<b><font color="#0000FF">def</font></b> value <b><font color="#0000FF">=</font></b> e1<font color="#990000">.</font>value <font color="#990000">+</font> e2<font color="#990000">.</font>value
<b><font color="#0000FF">case</font></b> <b><font color="#0000FF">class</font></b> <font color="#008080">Sub</font><font color="#990000">(</font>e1<b><font color="#0000FF">:</font></b> Expr<font color="#990000">,</font> e2<b><font color="#0000FF">:</font></b> Expr<font color="#990000">)</font> <b><font color="#0000FF">extends</font></b> Expr
<b><font color="#0000FF">def</font></b> value <b><font color="#0000FF">=</font></b> e1<font color="#990000">.</font>value <font color="#990000">-</font> e2<font color="#990000">.</font>value
<b><font color="#0000FF">case</font></b> <b><font color="#0000FF">class</font></b> <font color="#008080">Num</font><font color="#990000">(</font>n<b><font color="#0000FF">:</font></b> <font color="#009900">Int</font><font color="#990000">)</font> <b><font color="#0000FF">extends</font></b> Expr
<b><font color="#0000FF">def</font></b> value <b><font color="#0000FF">=</font></b> n
</tt></pre></blockquote>
<p>The object-oriented solution uses <em>subtype polymorphism</em>. We see that it is
easy to add new classes, such as a <code>Mul</code>, but if we try to add new function, we
have to change all the existing classes.</p>
<p>Here are the main points:</p>
<ul>
<li>Smart cases, i.e. Objects</li>
<li>Every case enumerates all functions</li>
<li>Very easy to add new cases</li>
<li>Very difficult to add new functions</li>
</ul>
<p>We get <em>a closed set of functions and an open set of cases!</em></p>
<h2>Dynamic Style</h2>
<p>Now lets solve it with Javascript in a dynamic style. The solution we have
looks a lot like the subtype polymorphic solution above.</p>
<blockquote><pre><tt><b><font color="#0000FF">function</font></b> <b><font color="#000000">Add</font></b><font color="#990000">(</font>e1<font color="#990000">,</font> e2<font color="#990000">)</font> <font color="#FF0000">{</font>
<b><font color="#0000FF">this</font></b><font color="#990000">.</font>e1 <font color="#990000">=</font> e1<font color="#990000">;</font>
<b><font color="#0000FF">this</font></b><font color="#990000">.</font>e2 <font color="#990000">=</font> e2<font color="#990000">;</font>
<font color="#FF0000">}</font>
Add<font color="#990000">.</font><b><font color="#0000FF">prototype</font></b><font color="#990000">.</font>value <font color="#990000">=</font> <b><font color="#0000FF">function</font></b><font color="#990000">()</font> <font color="#FF0000">{</font> <b><font color="#0000FF">return</font></b> <b><font color="#0000FF">this</font></b><font color="#990000">.</font>e1 <font color="#990000">+</font> <b><font color="#0000FF">this</font></b><font color="#990000">.</font>e2<font color="#990000">;</font> <font color="#FF0000">}</font><font color="#990000">;</font>
<b><font color="#0000FF">function</font></b> <b><font color="#000000">Sub</font></b><font color="#990000">(</font>e1<font color="#990000">,</font> e2<font color="#990000">)</font> <font color="#FF0000">{</font>
<b><font color="#0000FF">this</font></b><font color="#990000">.</font>e1 <font color="#990000">=</font> e1<font color="#990000">;</font>
<b><font color="#0000FF">this</font></b><font color="#990000">.</font>e2 <font color="#990000">=</font> e2<font color="#990000">;</font>
<font color="#FF0000">}</font>
Sub<font color="#990000">.</font><b><font color="#0000FF">prototype</font></b><font color="#990000">.</font>value <font color="#990000">=</font> <b><font color="#0000FF">function</font></b><font color="#990000">()</font> <font color="#FF0000">{</font> <b><font color="#0000FF">return</font></b> <b><font color="#0000FF">this</font></b><font color="#990000">.</font>e1 <font color="#990000">-</font> <b><font color="#0000FF">this</font></b><font color="#990000">.</font>e2<font color="#990000">;</font> <font color="#FF0000">}</font><font color="#990000">;</font>
<b><font color="#0000FF">function</font></b> <b><font color="#000000">Num</font></b><font color="#990000">(</font>n<font color="#990000">)</font> <font color="#FF0000">{</font>
<b><font color="#0000FF">this</font></b><font color="#990000">.</font>n <font color="#990000">=</font> n<font color="#990000">;</font>
<font color="#FF0000">}</font>
Num<font color="#990000">.</font><b><font color="#0000FF">prototype</font></b><font color="#990000">.</font>value <font color="#990000">=</font> <b><font color="#0000FF">function</font></b><font color="#990000">()</font> <font color="#FF0000">{</font> <b><font color="#0000FF">return</font></b> <b><font color="#0000FF">this</font></b><font color="#990000">.</font>n<font color="#990000">;</font> <font color="#FF0000">}</font><font color="#990000">;</font>
</tt></pre></blockquote>
<p>Just as in the polymorphic solution, it is easy to add a new class.</p>
<blockquote><pre><tt><i><font color="#9A1900">// Adding a new class</font></i>
<b><font color="#0000FF">function</font></b> <b><font color="#000000">Mul</font></b><font color="#990000">(</font>e1<font color="#990000">,</font> e2<font color="#990000">)</font> <font color="#FF0000">{</font>
<b><font color="#0000FF">this</font></b><font color="#990000">.</font>e1 <font color="#990000">=</font> e1<font color="#990000">;</font>
<b><font color="#0000FF">this</font></b><font color="#990000">.</font>e2 <font color="#990000">=</font> e2<font color="#990000">;</font>
<font color="#FF0000">}</font>
Mul<font color="#990000">.</font><b><font color="#0000FF">prototype</font></b><font color="#990000">.</font>value <font color="#990000">=</font> <b><font color="#0000FF">function</font></b><font color="#990000">()</font> <font color="#FF0000">{</font> <b><font color="#0000FF">return</font></b> <b><font color="#0000FF">this</font></b><font color="#990000">.</font>e1 <font color="#990000">*</font> <b><font color="#0000FF">this</font></b><font color="#990000">.</font>e2<font color="#990000">;</font> <font color="#FF0000">}</font><font color="#990000">;</font>
</tt></pre></blockquote>
<p>But, what about adding a new functions? It turns out that this is just as easy
because of the dynamic nature of Javascript. We just add them to the prototype.</p>
<blockquote><pre><tt><i><font color="#9A1900">// Adding new functions to existing prototypes</font></i>
Add<font color="#990000">.</font><b><font color="#0000FF">prototype</font></b><font color="#990000">.</font>toString <font color="#990000">=</font> <b><font color="#0000FF">function</font></b><font color="#990000">()</font> <font color="#FF0000">{</font>
<b><font color="#0000FF">return</font></b> <font color="#FF0000">'('</font> <font color="#990000">+</font> <b><font color="#0000FF">this</font></b><font color="#990000">.</font>e1<font color="#990000">.</font><b><font color="#000000">toString</font></b><font color="#990000">()</font> <font color="#990000">+</font> <font color="#FF0000">' + '</font> <font color="#990000">+</font> <b><font color="#0000FF">this</font></b><font color="#990000">.</font>e2<font color="#990000">.</font><b><font color="#000000">toString</font></b><font color="#990000">()</font> <font color="#990000">+</font> <font color="#FF0000">')'</font><font color="#990000">;</font>
<font color="#FF0000">}</font>
Sub<font color="#990000">.</font><b><font color="#0000FF">prototype</font></b><font color="#990000">.</font>toString <font color="#990000">=</font> <b><font color="#0000FF">function</font></b><font color="#990000">()</font> <font color="#FF0000">{</font>
<b><font color="#0000FF">return</font></b> <font color="#FF0000">'('</font> <font color="#990000">+</font> <b><font color="#0000FF">this</font></b><font color="#990000">.</font>e1<font color="#990000">.</font><b><font color="#000000">toString</font></b><font color="#990000">()</font> <font color="#990000">+</font> <font color="#FF0000">' - '</font> <font color="#990000">+</font> <b><font color="#0000FF">this</font></b><font color="#990000">.</font>e2<font color="#990000">.</font><b><font color="#000000">toString</font></b><font color="#990000">()</font> <font color="#990000">+</font> <font color="#FF0000">')'</font><font color="#990000">;</font>
<font color="#FF0000">}</font>
Num<font color="#990000">.</font><b><font color="#0000FF">prototype</font></b><font color="#990000">.</font>toString <font color="#990000">=</font> <b><font color="#0000FF">function</font></b><font color="#990000">()</font> <font color="#FF0000">{</font>
<b><font color="#0000FF">return</font></b> <b><font color="#0000FF">this</font></b><font color="#990000">.</font>n<font color="#990000">;</font>
<font color="#FF0000">}</font>
Mul<font color="#990000">.</font><b><font color="#0000FF">prototype</font></b><font color="#990000">.</font>toString <font color="#990000">=</font> <b><font color="#0000FF">function</font></b><font color="#990000">()</font> <font color="#FF0000">{</font>
<b><font color="#0000FF">return</font></b> <font color="#FF0000">'('</font> <font color="#990000">+</font> <b><font color="#0000FF">this</font></b><font color="#990000">.</font>e1<font color="#990000">.</font><b><font color="#000000">toString</font></b><font color="#990000">()</font> <font color="#990000">+</font> <font color="#FF0000">' * '</font> <font color="#990000">+</font> <b><font color="#0000FF">this</font></b><font color="#990000">.</font>e2<font color="#990000">.</font><b><font color="#000000">toString</font></b><font color="#990000">()</font> <font color="#990000">+</font> <font color="#FF0000">')'</font><font color="#990000">;</font>
<font color="#FF0000">}</font>
</tt></pre></blockquote>
<p>Now getting a string representation of an expression is a simple as:</p>
<blockquote><pre><tt><b><font color="#0000FF">var</font></b> x <font color="#990000">=</font> <b><font color="#0000FF">new</font></b> <b><font color="#000000">Num</font></b><font color="#990000">(</font><font color="#993399">1</font><font color="#990000">);</font>
<b><font color="#0000FF">var</font></b> y <font color="#990000">=</font> <b><font color="#0000FF">new</font></b> <b><font color="#000000">Num</font></b><font color="#990000">(</font><font color="#993399">2</font><font color="#990000">);</font>
<b><font color="#0000FF">var</font></b> z <font color="#990000">=</font> <b><font color="#0000FF">new</font></b> <b><font color="#000000">Add</font></b><font color="#990000">(</font>x<font color="#990000">,</font> y<font color="#990000">);</font>
<b><font color="#0000FF">var</font></b> w <font color="#990000">=</font> <b><font color="#0000FF">new</font></b> <b><font color="#000000">Sub</font></b><font color="#990000">(</font>x<font color="#990000">,</font> y<font color="#990000">);</font>
<b><font color="#0000FF">var</font></b> e <font color="#990000">=</font> <b><font color="#0000FF">new</font></b> <b><font color="#000000">Mul</font></b><font color="#990000">(</font>z<font color="#990000">,</font> w<font color="#990000">);</font>
e<font color="#990000">.</font><b><font color="#000000">toString</font></b><font color="#990000">();</font> <i><font color="#9A1900">// returns ((1 + 2) * (1 - 2))</font></i>
</tt></pre></blockquote>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjHehhFgxXzzv5gJ9ciQh3Y1yI4k3rZgYGFrIehfEvn2S01xIrp3TqvfcIvig8fjAZl1oXM3XzdmY6b1ms4XqRlYJkHWgTLWxSSNGiktMYuvFsB39x6ROCzab3WVSqrf7EBxuR4/s1600/CayenneChilePepper.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjHehhFgxXzzv5gJ9ciQh3Y1yI4k3rZgYGFrIehfEvn2S01xIrp3TqvfcIvig8fjAZl1oXM3XzdmY6b1ms4XqRlYJkHWgTLWxSSNGiktMYuvFsB39x6ROCzab3WVSqrf7EBxuR4/s200/CayenneChilePepper.jpg" /></a></div>
Well, isn't that nice!
<blockquote>
<em>
Sometimes I feel like I don't have a problem<br/>
...<br/>
I don't ever feel like I did before<br/>
But take me to a place I love, a dynamic place!<br/>
I don't ever feel like I did before<br/>
But take me to a place I love, a dynamic place, yeah, yeah, yeah!<br/>
<strong>Misquoting Red Hot Chili Peppers :)</strong>
</em>
</blockquote>
Anders Janmyrhttp://www.blogger.com/profile/16081537018884349622noreply@blogger.com7tag:blogger.com,1999:blog-34049130.post-34682528514040555822013-05-24T08:41:00.000+02:002013-06-03T10:19:00.336+02:00A Critique of the Thoughtworks Tech Radar on Javascript Testing<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiyXj0ROap9mS8-tBc2ZXVPsScnvqhyphenhyphentfqSZ4gbZSjZsQl5YwZ80hviFxHgOFg4LTmOpgoU4oQp0hsey3uC8drbYZkjldI2oAg_yynLJNTuSLc0Je2NDY1slvknjhPRYNi2COqx/s1600/Immanuel_Kant.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiyXj0ROap9mS8-tBc2ZXVPsScnvqhyphenhyphentfqSZ4gbZSjZsQl5YwZ80hviFxHgOFg4LTmOpgoU4oQp0hsey3uC8drbYZkjldI2oAg_yynLJNTuSLc0Je2NDY1slvknjhPRYNi2COqx/s320/Immanuel_Kant.jpg" /></a>
<p>The <a href="http://www.thoughtworks.com/radar">Thoughtworks' Tech Radar</a> has come out
again and there is no change in the recommendation on Javascript testing.</p>
<p>The radar recommends to "Adopt Jasmine paired with Node.js". This is very
specific advice. It's not "Adopt Javascript testing paired with Node.js" but a
specific tool, Jasmine. Compare this with more general advice such as "Adopt
CSS Frameworks" or "Promises for asynchronous programming". Nothing specific
there and, hence, nothing wrong.</p>
<p>There is no motivation as to why we should adopt Jasmine but, if we look at the
<a href="http://thoughtworks.fileburst.com/assets/technology-radar-october-2012.pdf">TTF from October 2012</a>(PDF),
we can read:</p>
<blockquote>The cream of the crop for out-of-browser testing is currently Jasmine. Jasmine
paired with Node.js is the go-to choice for robust testing of both client- and
serverside JavaScript.</blockquote>
<p>"The cream of the crop"? This is certainly not the case! Jasmine is an elegant
testing framework similar Ruby's RSpec but, it is <em>not</em> "the cream of the
crop"! It has one major drawback. It sucks at testing asynchronous code! And,
since asynchronous code is a central theme in Javascript programming we should
not use Jasmine! There are at least two better alternatives:</p>
<ul>
<li><a href="http://docs.busterjs.org/">Buster.js</a></li>
<li><a href="http://visionmedia.github.com/mocha/">Mocha</a></li>
</ul>
<h3>Example</h3>
<p>In this example I want to test a simple asynchronous sort function called
<code>sleepsort</code>, you can read more about it in <a href="http://anders.janmyr.com/2012/04/writing-node-module.html">Writing a Module</a>.</p>
<p>The function signature looks like this:</p>
<blockquote><pre><tt><i><font color="#9A1900">// Sleepsort takes an array of numbers and calls callback(result)</font></i>
<i><font color="#9A1900">// with the sorted array as the only argument</font></i>
<b><font color="#0000FF">function</font></b> <b><font color="#000000">sleepsort</font></b><font color="#990000">(</font>array<font color="#990000">,</font> callback<font color="#990000">)</font> <font color="#FF0000">{</font>
<font color="#990000">...</font>
<font color="#FF0000">}</font>
</tt></pre></blockquote>
<p>Testing this with Jasmine will look something like this:</p>
<blockquote><pre><tt><b><font color="#000000">it</font></b><font color="#990000">(</font><font color="#FF0000">'sorts the array'</font><font color="#990000">,</font> <b><font color="#0000FF">function</font></b> <font color="#990000">()</font> <font color="#FF0000">{</font>
<b><font color="#000000">runs</font></b><font color="#990000">(</font><b><font color="#0000FF">function</font></b><font color="#990000">()</font> <font color="#FF0000">{</font>
<b><font color="#0000FF">this</font></b><font color="#990000">.</font>result<font color="#990000">;</font>
<b><font color="#000000">sleepsort</font></b><font color="#990000">([</font><font color="#993399">1</font><font color="#990000">,</font> <font color="#993399">3</font><font color="#990000">,</font> <font color="#993399">2</font><font color="#990000">,</font> <font color="#993399">7</font><font color="#990000">],</font> <b><font color="#0000FF">function</font></b><font color="#990000">(</font>result<font color="#990000">)</font> <font color="#FF0000">{</font>
<b><font color="#0000FF">this</font></b><font color="#990000">.</font>result <font color="#990000">=</font> result<font color="#990000">;</font>
<font color="#FF0000">}</font><font color="#990000">);</font>
<font color="#FF0000">}</font><font color="#990000">);</font>
<b><font color="#000000">waits</font></b><font color="#990000">(</font><font color="#993399">1000</font><font color="#990000">);</font> <i><font color="#9A1900">// Wait one second then check the rsult</font></i>
<b><font color="#000000">runs</font></b><font color="#990000">(</font><b><font color="#0000FF">function</font></b> <font color="#990000">()</font> <font color="#FF0000">{</font>
<b><font color="#000000">expect</font></b><font color="#990000">(</font><b><font color="#0000FF">this</font></b><font color="#990000">.</font>result<font color="#990000">).</font><b><font color="#000000">toEqual</font></b><font color="#990000">([</font><font color="#993399">1</font><font color="#990000">,</font> <font color="#993399">2</font><font color="#990000">,</font> <font color="#993399">3</font><font color="#990000">,</font> <font color="#993399">7</font><font color="#990000">]);</font>
<font color="#FF0000">}</font><font color="#990000">);</font>
<font color="#FF0000">}</font><font color="#990000">);</font>
</tt></pre></blockquote>
<p>This is horrible! Compare this to Mocha, with <code>should.js</code> assertions:</p>
<blockquote><pre><tt><b><font color="#000000">it</font></b><font color="#990000">(</font><font color="#FF0000">'Sorts the array'</font><font color="#990000">,</font> <b><font color="#0000FF">function</font></b><font color="#990000">(</font>done<font color="#990000">)</font> <font color="#FF0000">{</font>
<b><font color="#000000">sleepsort</font></b><font color="#990000">([</font><font color="#993399">1</font><font color="#990000">,</font> <font color="#993399">3</font><font color="#990000">,</font> <font color="#993399">2</font><font color="#990000">,</font> <font color="#993399">7</font><font color="#990000">],</font> <b><font color="#0000FF">function</font></b><font color="#990000">(</font>result<font color="#990000">)</font> <font color="#FF0000">{</font>
result<font color="#990000">.</font>should<font color="#990000">.</font><b><font color="#000000">eql</font></b><font color="#990000">([</font><font color="#993399">1</font><font color="#990000">,</font> <font color="#993399">2</font><font color="#990000">,</font> <font color="#993399">3</font><font color="#990000">,</font> <font color="#993399">7</font><font color="#990000">]);</font>
<b><font color="#000000">done</font></b><font color="#990000">();</font>
<font color="#FF0000">}</font><font color="#990000">);</font>
<font color="#FF0000">}</font><font color="#990000">);</font>
</tt></pre></blockquote>
<p>Thank you, Sir, may I have another!</p>
<p>Indeed you may, here it is with Buster.js, with Object style syntax:</p>
<blockquote><pre><tt><font color="#FF0000">"Sorts the array"</font><font color="#990000">:</font> <b><font color="#0000FF">function</font></b> <font color="#990000">(</font>done<font color="#990000">)</font> <font color="#FF0000">{</font>
<b><font color="#000000">sleepsort</font></b><font color="#990000">([</font><font color="#993399">1</font><font color="#990000">,</font> <font color="#993399">3</font><font color="#990000">,</font> <font color="#993399">2</font><font color="#990000">,</font> <font color="#993399">7</font><font color="#990000">],</font> <b><font color="#0000FF">function</font></b><font color="#990000">(</font>result<font color="#990000">)</font> <font color="#FF0000">{</font>
assert<font color="#990000">.</font><b><font color="#000000">equals</font></b><font color="#990000">(</font>result<font color="#990000">,</font> <font color="#990000">[</font><font color="#993399">1</font><font color="#990000">,</font> <font color="#993399">2</font><font color="#990000">,</font> <font color="#993399">3</font><font color="#990000">,</font> <font color="#993399">7</font><font color="#990000">]);</font>
<b><font color="#000000">done</font></b><font color="#990000">();</font>
<font color="#FF0000">}</font><font color="#990000">);</font>
<font color="#FF0000">}</font>
</tt></pre></blockquote>
<p>Both Mocha and Buster.js send in a <code>done</code> function to call when the execution
has finished. Elegant, and above all, crystal clear.</p>
<p>It is very obvious to me that Jasmine no longer is "the cream of the crop" of
Javascript testing. But, what do I know, perhaps it was a copy-paste error.</p>
<p>I've done my duty, now I can go to bed!</p>
<div style="margin-left: 1em; margin-right: 1em;">
<img alt="Duty Calls from XKCD" src="http://imgs.xkcd.com/comics/duty_calls.png" />
</div>
Anders Janmyrhttp://www.blogger.com/profile/16081537018884349622noreply@blogger.com0tag:blogger.com,1999:blog-34049130.post-938144511292351792013-04-30T14:00:00.000+02:002013-05-01T20:41:24.687+02:00A Responsible Programmer<div style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi_XdPMqc2iXWbmuYOMXUBCyEsepaRqtED1ORrlvEORl70JarKFXC0XBqsMVIcA8l7Z6eMqLF6R1puDNr7lkaksi4CD2oYPy3MpDmVKoXlNFIGUqDR0zI0Hek_kHun40bKtwIOc/s320/thumbs-up-smiley.jpg" /></div>
<p>
In the last few years I have been asked to help savor several web projects gone
bad. The quality of the projects, code, environment, documentation, and morale
has been low. To know what I think is right in situations like this, I started
asking myself the question, "What would a responsible programmer do?".</p>
<h2>Clarity</h2>
<p>Above anything else a responsible programmer values clarity. Not only does she
value clear code, but also clear documentation, clear communication and a
clear vision of where she and her project is going.</p>
<h2>Coding</h2>
<h3>Write Consistent Code</h3>
<p>The responsible programmer writes consistent code. Consistency helps other
programmers read and understand her code. It lets them know what to expect. If
she names constants with <code>SCREAMING_SNAKE_CASE</code>, they know that they wont
change. When naming attributes in CSS and HTML, she will make all of them
<code>dash-er-ized</code>, or none of them. This is easy stuff, but important. Consistency
breeds familiarity. Familiarity is good, it removes worry and increases
confidence.</p>
<p>When the responsible programmer contributes code to other projects, she will
make sure that she, consistently, follows the style of the project. Sometimes
it is not easy to tell what style a project uses, the responsible way is to ask
what style is preferred and then use that style. By just asking the question,
she will often trigger a review of the code which will help set a consistent
style in the future.</p>
<p>By writing consistent code, a responsible programmer will make the program
easier to understand and easier to maintain.</p>
<h3>Don't Quick-Fix</h3>
<p>A responsible programmer doesn't do quick fixes. When a bug needs to be fixed
she fixes the root problem instead of fixing the symptom. If an event-handler
suddenly starts receiving unnamed events the proper fix is not to ignore
unnamed events but, to figure out why unnamed events come at all when not
expected. She knows that fixing a symptom will only make the root cause much
harder to find.</p>
<h3>Write Short Functions</h3>
<p>Short functions are easier to understand, easier to reason about and easier to
test. It is the responsible thing to write. Enough said!</p>
<h3>Separate Commands From Queries</h3>
<p>CQS or Command Query Separation has become all the rave in the DDD world, but
it was coined by Bertrand Meyer in the book,
<a href="http://www.amazon.com/Object-Oriented-Software-Construction-Book-CD-ROM/dp/0136291554?tag=thtasta-20">Object-Oriented Software Construction</a> in 1988.
A good book, read it!</p>
<p>The responsible programmer separates her commands from her queries because she
knows that they are easier to test and that she can call the queries many times
without anything bad happening.
Separating your commands form your queries may be as easy as:</p>
<blockquote><pre><tt><i><font color="#9A1900">// Can be called whenever, no side-effects</font></i>
<b><font color="#0000FF">function</font></b> <b><font color="#000000">generateRoute</font></b><font color="#990000">(</font>params<font color="#990000">)</font> <font color="#FF0000">{</font>
<b><font color="#0000FF">return</font></b> <font color="#990000">[</font>params<font color="#990000">.</font>major<font color="#990000">,</font> params<font color="#990000">.</font>minor<font color="#990000">,</font> params<font color="#990000">.</font>patch<font color="#990000">].</font><b><font color="#000000">join</font></b><font color="#990000">(</font><font color="#FF0000">'/'</font><font color="#990000">);</font>
<font color="#FF0000">}</font>
<i><font color="#9A1900">// Updates the hash with the new route.</font></i>
<b><font color="#0000FF">function</font></b> <b><font color="#000000">updateRoute</font></b><font color="#990000">(</font>params<font color="#990000">)</font> <font color="#FF0000">{</font>
location<font color="#990000">.</font>hash <font color="#990000">=</font> <b><font color="#000000">generateRoute</font></b><font color="#990000">(</font>params<font color="#990000">);</font>
<font color="#FF0000">}</font>
</tt></pre></blockquote>
<h3>Refactor Mercilessly</h3>
<p>Since a responsible programmer values clarity, she refactors mercilessly when
her understanding of the system changes. She knows that the time invested in
making the code a little more clear will prevent bugs and frustration in the
future.</p>
<h3>Prefer Explicit</h3>
<p>A responsible programmer prefers explicit code over implicit code. Even though
her understanding of advanced concepts such as meta-programming, monads and
continuations is substantial, she prefers explicit code over beautiful
abstractions. New programmers (including her future self) have a lot easier to
understand code that is explicit than code that is not.</p>
<h3>Don't Fear Advanced Techniques</h3>
<p>Since advanced techniques can make the code a lot simpler in certain
situations she never shies away from advanced techniques when she realizes
that they are called for. At the end of the day meta-programming and "advanced"
functional programming techniques are just tools that should be used when
appropriate.</p>
<h3>Check Boundaries</h3>
<p>A responsible programmer always checks the boundaries of her system to make
sure that invalid data doesn't enter into the core of the application. This way
she can avoid defensive programming in the core domain where clarity is even
more essential than anywhere else.</p>
<h3>Wrap External Services</h3>
<p>External services is one of the main reasons that development takes time,
the responsible programmer makes sure to always wrap external services with a
local interface. This simplifies both testing and exchanging the services.</p>
<h3>External Libraries</h3>
<p>A responsible programmer will never use an external library she doesn't trust.
She will never add a library into her code base unless there is a significant
reason for adding it. When she adds an external library she will learn it. She
will learn how she configures it, how it is to be called, what are good
practices for using it, what bugs there are, etc.</p>
<h3>Balance</h3>
<p>A code base can be compared to a balanced tree. A balanced tree is
data-structure that will rebalance itself when new items are added to it. This
makes the cost of modifying the tree more expensive but has the benefit that
accessing items in the tree can be performed in an optimal way.</p>
<p>A responsible programmer treats her code base as a balanced tree. She will
never add code without thinking about balance. She knows that if the
project loses its balance it may have to be entirely rewritten to regain
balance again.</p>
<p>The balance of the code may shift as the code matures, when a major shift is
called for the responsible programmer will refactor mercilessly to obtain a new
optimal balance.</p>
<h2>Documentation</h2>
<p>A responsible programmer writes and <em>maintains</em> documentation as the needs come
up. The needs differ between projects, but most projects benefit from a system
overview, a domain overview, a style guide, and code comments.</p>
<p>A responsible programmer makes assumptions all the time when coding, she writes
the assumptions as comments in the code when she makes them. She tags them
to make it possible to generate a list of assumptions.</p>
<h3>The System Overview</h3>
<p>The system overview is a drawing and a description of all the servers that are
involved in the system. This includes databases, queues, web-servers, external
services, etc. The description describes how the pieces fit together.</p>
<h3>The Domain Overview</h3>
<p>This is a drawing and a high level description of how the core domain of the
system works. It includes the major concepts of the domain and what they mean.</p>
<h3>The Style Guide</h3>
<p>The style guide may be as easy as referring to <a href="https://github.com/styleguide/javascript">Github's style guide</a> or to write <a href="http://anders.janmyr.com/2012/03/braceless-programming.html">your own</a> that alters someone else's. Anyway you do it, it is worth having it written down.</p>
<h3>Code Comments</h3>
<p>Comments in code should be very sparse and only added to point out
idiosyncrasies. When assumptions are made, they can be written as comments with
a tag to make it possible to generate a list. Example:</p>
<blockquote><pre><tt><i><font color="#9A1900">// ASSUMPTION: The list is expected to be small and will</font></i>
<i><font color="#9A1900">// be entirely loaded from the server</font></i>
<b><font color="#0000FF">function</font></b> <b><font color="#000000">loadCities</font></b><font color="#990000">()</font> <font color="#FF0000">{</font>
<font color="#FF0000">}</font>
</tt></pre></blockquote>
<h2>Testing</h2>
<p>The responsible programmer tests! She doesn't test for the sake of testing or
to increase code coverage, she tests to be sure that the code works as she
expects it to.</p>
<p>She knows that dynamic environments such as Javascript and web browsers are
fragile and that it is easy to break code without meaning to.</p>
<p>How to write good tests have been written about elsewhere and I wont spend any
time on it here. I can recommend the last chapter in Sandi Metz' book, <a href="http://www.amazon.com/dp/0321721330">Practical Object-Oriented Design in Ruby</a> if you are interested in good techniques for testing in dynamic programming languages.</p>
<h2>Environments</h2>
<p>The responsible programmer owns her environments. In a project there are at
least three environments to care about, Production, Test, and Development.
There is also the development machine itself.</p>
<p>The responsible programmer can set up all project environments with a single
command that installs everything that is needed, including databases, seed
data, libraries, search engines, tools, environment variables, SSH-keys.
Everything!</p>
<p>This will allow a new programmer starting in the project to be setup within
minutes. It will also allow her to experiment with anything without having to
worry about destroying the setup and losing days debugging the environment.</p>
<p>Her personal development machine is also perfectly configured at all times. If
she learns a new trick, she will immediately incorporate it into her toolset
and into her configuration.</p>
<p>She is automatically prepared for catastrophes. If her hard disk crashes she
can just buy a new one at the local supermarket and install her configuration
files and be ready to go within hours.</p>
<p>To make this happen she always keeps backups of her configuration files. She
keeps the non-secret ones on Github and the secret ones, such as SSH keys and
passwords elsewhere.</p>
<h3>Continuous Integration/Deployment</h3>
<p>Another part of the environment is continuous integration. If a project doesn't
use continuous integration it is a clear sign that it is not healthy.</p>
<p>The continuous integration server is just another environment and setting up a
new is done with a single command just like the others.</p>
<p>The responsible programmer will set up and maintain continuous integration just
like she does every other environment.</p>
<h3>Scripting</h3>
<p>In order to achieve the environment goals, a responsible programmer knows how
to script. Scripting is not only essential for keeping your environments up to
date, they are essential for automating simple tasks. Scripts are useful for
generating code, testing, refactoring, renaming, installation, automating
checklists, etc.</p>
<p>A responsible programmer ask herself, "When did I do something once? Never!"
Writing a script that does what she wants frees her from having to remember the
sequence of instructions required to do a task and let's her focus on more
important stuff. It also serves as runnable documentation.</p>
<h2>Tools</h2>
<p>A responsible programmer knows her tools. She will always try to learn more
about them and she will replace them if other tools are invented that work
better. But she doesn't change her tools for the latest fashion.</p>
<p>The command line is a very powerful tool and so is scripting language and
a scriptable editor.</p>
<h3>Version Control</h3>
<p>The responsible programmer uses version control to communicate with her future
self and with other programmers. She knows that a clear commit message will
help her and others understand what has happened to the system.</p>
<p>She prunes her commits. When she has made several changes to a code base she
will make sure that she commits the different changes separately by using
something like <code>git add --patch</code>. She also knows that if she commits something
by mistake she can alter the commit message or add forgotten files with <code>git
commit --amend</code> and that she can change the contents of her history with <code>git
rebase --interactive</code> or with <code>git reset</code></p>
<h2>Projects</h2>
<h3>Own It</h3>
<p>A responsible programmer owns her project, she will not allow anything bad to
happen to her code. This is a difficult goal to achieve when she comes into a
project that has already gone bad, but it is a worthy goal and not one that
should be taken lightly. All projects must have at least one person who owns
the code. When people start talking about the code as if it is someone else's,
it is time to shut the project down.</p>
<p>If a responsible programmer decides to take ownership of a project, she makes
sure that she has the authority to make the decisions that she deems necessary.
No authority, no responsibility, it's as simple as that.</p>
<h3>Estimation</h3>
<p>Sometimes projects require estimates, most of the time they don't but,
sometimes they actually do. A responsible programmer knows how to estimate. She
knows that an estimate is just a guess and the bounds of any task, however
trivial, always have a small probability of taking infinitely long to finish
(earth quake, meteor strike, blackout). There is also a small probability that
the code is already being written at the time of estimation.</p>
<p>Being aware that the code may take infinitely long to finish she is very
careful not to make any promises and she is very clear about her estimates
being guesses.</p>
<h3>Don't Do as They Are Told</h3>
<p>Some people may not see this as a sign of a responsible programmer but I beg to
differ. When a programmer is told to do something, she will try to figure out
what the real problem is. She may do this in several different ways. She may
sit down and think through the "task" and come up with an alternate solution
that solves the problem simpler or, even better, makes the problem go a way
completely. She may ask questions to help clarify the problem for her. Why is
this a problem? Why do you do it like this? Why? Why? Why?</p>
<p>Some people may not like this and tell her to "Just fucking do it!". Her reply to
this is something along the lines of "Just fucking do it yourself!" but she is
usually a lot more polite so she may very well say "I don't understand what
your problem is and, therefore, I am not the best person to solve it, please ask
someone else."</p>
<p>She believes it is her job to understand what she is doing and why, and that
life is too short to be a drone!</p>
<p>If she feels that she is not able to be a responsible programmer on a project,
the responsible thing to do is to leave.</p>
<h2>Summary</h2>
<p>I have found that asking myself the question "What would a responsible
programmer do?" liberating. It clarifies what I should do in situations of
doubt.</p>
<p>At the end of the day, the responsible programmer can look through the commit
log and see a beautiful list of tasks that she has completed. She can look
through each commit and see that they are cohesive and well described by their
commit message. She can <code>git blame</code> the code and see that every line of code
that has her name on it reads well. She can look at her days work, and she can
feel proud!</p>
Anders Janmyrhttp://www.blogger.com/profile/16081537018884349622noreply@blogger.com28