MySQL very high CPU usage (and other processes)

Hi everyone

TL;DR (or Summary) – Leap second added on 30 June 2015 caused MySQL (and other processes to have very high CPU usage)

This is just a quick post on an issue that I faced today (01/07/2015). I arrived at work and found a warning from our monitoring tool that we had high cpu usage beginning at 01.00 AM. On top we saw several processes using a very high amount of CPU, namely:

  • mysql
  • jenkins
  • tomcat
  • ruby (god)
  • ksoftirqd (several of these)

On Tomcat we can have legitimate high cpu usage, but jenkins was not building anything and mysql (version 5.5.31) was not used in anything important. We ran through the logs and analytics to check if we were having an abnormally high visitor access and found nothing, nada, zero. We logged into mysql and issued the SHOW PROCESSLIST command and nothing was happening.

ksoftird was particularly weird, because it is “is a per-cpu kernel thread that runs when the machine is under heavy soft-interrupt load” which didn’t make any sense to us.

What the…. ? (X-Files theme song playing in the background)

After a certain time and more attempts (like trying to use jstack on tomcat to see if it showed anything unusual) we decided to restart mysql, tomcat, jenkins and god, but it all went back again to high cpu usage. We even restarted the server without better luck.

We then read this stack overflow question more carefully and noticed the “leap second issue”, and someone said “hey I remember seeing something about a leap second not to long ago”… I went and check wikipedia, and sure a leap second was added on June 30th 2015.

We tried the following:

$ /etc/init.d/ntpd stop
$ date -s "`date`"
$ /etc/init.d/ntpd start

Some people said that restarting ntp was not required, but… we did it anyway :) And, it worked. Magically everything went back to normal. Apparently we weren’t the only ones affected. I think the next time I see a leap second announcement I’m going to put it on my calendar and warning to a day before :)

Mozilla’s blog also documents the issue.

I know this issue is documented, but I wanted to have one more post contributing to people finding this stuff on the internet, could save someone a couple of hours next time.

Happy coding and stay safe of leap seconds!

P.S – Reddit comments over here, Hacker News comments over here


I’m going on an adventure

Me, going on an adventure

Me, going on an adventure

I’ve been developing mostly back-end code for most of my career (Java essentially), I dealt with Java the language, the JVM, Eclipse, Java Enterprise, Databases (Oracle, MySQL, SQL Server), XML, XML Schema, etc… and I did some front-end hacking while developing some of the widgets of the XEO Framework.

I decided that I needed to do something different. I’ve always been fascinated with the front-end world (not necessarily thinking I should have done that all along, I very much like to do back-end code). I’d been reading on all the changes happening in front-end land, such as the MV* frameworks like Angular and Ember, the “recent” appearing of React (and React Native), all the fuss about node.js (io.js), all the tools like bower, gulp, grunt, bootstrap, responsive, rendering performance… a whole new world.

I’ve been fortunate enough to find a new challenge on Memeoirs where I’ve been given the task of managing front-end tasks (including our Ember.JS + EmberCLI application).

What I can say is that it’s been a very humbling experience: In a Java environment I know my way around everything, I know where to search, I’m very productive with Java. On the other hand, working mostly on Javascript for the better part of my day (and using tools like the ember-cli, or having to create my own build process using Broccoli for instance) reminds me that I’m in an incredibly huge field where there is so much stuff that you can do and learn. I’ll still have to learn CSS properly (I hope), responsive design, mockups, even my own text editor changed!

I expect an incredible amount of new things to learn, a lot of walls to bang my head against, but I think It’ll do me very good to have stepped out of my comfort zone and try new things. I believe this knowledge will be very helpful on my future (whatever that may be, because as Steve Jobs said, you can’t connect the dots while looking forward).

Happy coding!

If enterprise applications were made as hobby projects

Hi everyone, really long time no see…. sorry about that.

I saw a post the other day about why enterprise software sucks and god knows I agree with it, most enterprise software sucks for the reasons Jarkko outlines, but one thing that came to my mind was that in the end… the developers don’t relate to the problem at all. (Among other things of course).

See, why do we get so many open source projects like mvc frameworks, libraries, functional languages, web servers, css frameworks? Each one claiming that it’s really a change in the way you code something and that you can do better and faster. (I’m not saying they’re not, that’s not the point, just that there are lots of them, which is very cool)

You don’t see the same amount of open source projects to help mechanical engineers or gardeners, why? Well, because we’re not mechanical engineers nor gardeners, plain simple. We’re software developers and the pains we understand the most are our own. That’s why we spend our free time creating these frameworks and libraries, each one with a more convenient API, better performance and easier to use (and hopefully better documentation) to solve our own problems and sometimes you solve the problems of other people as well, hence the open-source projects. And the community is very vibrant, almost every day on Hacker News I see some new project being announced (especially in Javascript, it seems Jeff Atwood was right after all – any application that can be written in JavaScript, will eventually be written in JavaScript)

If you create a home made application to manage your expenses, you do it so that it’s easy to use and quick! (you don’t want to waste an entire afternoon just to add a few expenses, right?). You give it a lot of thought on how to improve your life (be it a small gain or a bigger one) because you know what pains you and you know how to solve that problem. It may not be the most usable application for everyone but it is to you, and you are your client.

In an enterprise context, you have no fraking clue of the problems people face, because (obviously) you’re not them. A more realistic scenario is the project manager talks to someone who pays for the product and explains to the manager what the application needs to do… and we implement it, without the least amount of knowledge of how the application should really be and how the users will use them, worse than that, we don’t know how they would like to use it.

The problem is, you can’t expect non-technical people to know what their needs are. Well, let me explain better… these people know what their problem is, and they know that they want to solve it, the problem is the HOW part. And I believe it’s our responsibility to find out what they need in order to solve their problem.

The whole agile movement, with its short iterations, scrum meetings and scrum masters, stakeholders, etc… allows you to see there’s a problem earlier and fix it with less cost, but you don’t solve anything if you’re talking to wrong stakeholder.

That’s why, I believe, we need need developers to get out and see HOW people work, what their pain is, and then create something like it was your own hobby project where you do things  to make your like easier, that’s what these people want as well, an application that helps them during their day job, not something to curse about. We need to sit by people’s side and learn how they perform their job, so that we can come up with good solutions!

I know that are a lot for issues with this approach, beginning with the fact that it goes against established procedures and although most of my experience is creating developers tools, I’ve been doing some side-projects that require just that, to know someone’s problem and help them solve it with a web application and it’s really been a fun ride. I must say that rapid prototyping really really helps out, because as soon as people start seeing things, they find out what they don’t want really quick and also being able to spend some time with them and understanding their issues is also really helpful, because you can see what they need (even when sometimes they don’t realize it themselves).

Any thoughts? Share them in the comments

Happy coding!