Modern Knowledge Blog

Recently I have been working on a few Angular applications and one of the problems I have been challenged with is styling an upload button and having it reliably post up to a Java Multipart Form Post. I have pulled out the useful bits of this approach and wanted to share them in case others find it useful.

All of the code is available on GitHub and hopefully there are good enough instructions in the README to help anyone set it up and run it.

For those that care I used Dropwizard as the backend Java service layer as it provided a quick and easy way to put together a layer to facilitate backend services. If you haven’t seen Dropwizard it was created by Coda Hale at Yammer and is constantly going through updates. It provides a best of breed conglomeration of Java libraries (Jersey, Jackson, and JDBI to name a few).

Here is the simple service that allows us to save the files that are uploaded.

 
package com.neosavvy;
 
import com.neosavvy.model.Attachment;
import com.sun.jersey.core.header.FormDataContentDisposition;
import com.sun.jersey.multipart.FormDataParam;
import com.yammer.dropwizard.logging.Log;
 
import javax.ws.rs.*;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import java.io.*;
 
@Path("/backend")
public class AttachmentResource {
 
    private static final Log LOG = Log.forClass(AttachmentResource.class);
 
    private String uploadDirectory;
 
    public AttachmentResource(String uploadDirectory) {
        super();
        this.uploadDirectory = uploadDirectory;
    }
 
    @POST
    @Path("/uploadattachment")
    @Consumes(MediaType.MULTIPART_FORM_DATA)
    public void uploadFile(
            @QueryParam("apiKey") String apiKey,
            @FormDataParam("file") InputStream uploadedInputStream,
            @FormDataParam("file") FormDataContentDisposition fileDetail) {
 
        final Attachment attachment;
 
        try {
 
            String uploadedFileLocation = "";
 
            ensureParentDirectory(uploadDirectory);
 
            uploadedFileLocation = fileDetail.getFileName().toLowerCase();
 
            // save it
            String uploadedFileLocationToWrite = uploadDirectory + uploadedFileLocation;
            writeToFile(uploadedInputStream, uploadedFileLocationToWrite);
 
        } catch (WebApplicationException e) {
            if (e.getResponse().getStatus() == Response.Status.UNSUPPORTED_MEDIA_TYPE.getStatusCode()) {
                LOG.error("The user attempted to upload a file that isn't supported ");
            } else {
                LOG.error("Unhandled exception occurred: " + e.getResponse().getStatus());
            }
            LOG.error("ERROR UPLOADING ATTACHMENT ");
 
            throw new WebApplicationException(Response.Status.INTERNAL_SERVER_ERROR);
        } catch (Exception e) {
            LOG.error("ERROR UPLOADING ATTACHMENT FOR UNKNOWN REASON");
            throw new WebApplicationException(Response.Status.INTERNAL_SERVER_ERROR);
        }
    }
 
    private void ensureParentDirectory(String parentDirectory) {
        File parentDir;
        if (parentDirectory != null) {
            parentDir = new File(parentDirectory);
            if (!parentDir.exists()) {
                parentDir.mkdirs();
            }
        } else {
            throw new WebApplicationException(Response.Status.PRECONDITION_FAILED);
        }
    }
 
    private void writeToFile(InputStream uploadedInputStream,
                             String uploadedFileLocation) {
        try {
            OutputStream out;
            int read = 0;
            byte[] bytes = new byte[1024];
 
            out = new FileOutputStream(new File(uploadedFileLocation));
            while ((read = uploadedInputStream.read(bytes)) != -1) {
                out.write(bytes, 0, read);
            }
            out.flush();
            out.close();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

Now with respect to Angular there are a few parts that make this work. One is a directive that can be used to associate any element styled in any way with a normal file upload input tag. Then it is just positioned in front of the element that has been styled and set to an opacity of 0 so the user can’t see it. However it will collect all the clicks from the user.

angular.module('upload-app', ['upload-app.services']).
  directive('fileButton', function() {
    return {
      link: function(scope, element, attributes) {
 
        var el = angular.element(element)
        var button = el.children()[0]
 
        el.css({
          position: 'relative',
          overflow: 'hidden',
          width: button.offsetWidth,
          height: button.offsetHeight
        })
 
        var fileInput = angular.element('<input id="uploadInput" type="file" multiple />')
        fileInput.css({
          position: 'absolute',
          top: 0,
          left: 0,
          'z-index': '2',
          width: '100%',
          height: '100%',
          opacity: '0',
          cursor: 'pointer'
        })
 
        el.append(fileInput)
 
 
      }
    }
  }).run();

To use it on a page I have implemented an AttachmentController to handle the actions of the UI but the HTML simply has a button on it using Bootstrap to style it. Then the file-button directive is activated on it’s parent div element. The HTML goes something like this:

        <div class="row span10" style="margin: 0px; padding-bottom: 10px" ng-controller="AttachmentController">
            <div class="span2" style="margin: 0px; ">
                <div style="float: left;">
                    <div id="dropbox" ng-class="dropbox">
                        <div file-button>
                            <button class='btn btn-success btn-large'><i class="icon-camera"></i></button>
                        </div>
                    </div>
                </div>
            </div>
            <div ng-show="files.length" class="span4">
                <button class='btn' ng-click="deleteCurrentAttachment()"><i class="icon-remove-circle"></i></button>
                <div ng-repeat="file in files">
                    <span>{{file.webkitRelativePath || file.name}}</span>
                </div>
            </div>
        </div>

Finally the click actions are handled in the controller as are the submissions to the service. The original example I followed is listed below as Drag and Drop File Upload. This was useful to show me how to use the basic hooks for uploading a multi-part mime post with an Angular controller. I was unable to make this work using the $http or $resource services in Angular and followed the example of a raw XHR request.

Hopefully this sample can help people understand this simple approach I am using and will be useful to others when they have similar challenges.

Sources

Styled Upload Button
Drag and Drop File Upload

Recently I have been working with a client who wants to migrate their primarily flash based marketing site over to a more HTML friendly solution. They want to do it in phases, but this has allowed me to take a nice look at the landscape of available JavaScript frameworks available for providing a practical approach to the development of the site and also other applications out there that I might get to work with.

I started by creating a simple Restful interface for trivially simple user objects. If you want to jump straight to the code it is available here

The interface was quite simple and just exposed simple PUT, DELETE, GET, and POST to support all the CRUD (Create, Read, Update and Delete) necessary for storing user data. The following code was what was defined using Jersey and DropWizard:

package com.neosavvy;
 
import com.neosavvy.model.User;
 
import javax.ws.rs.*;
import javax.ws.rs.core.MediaType;
import java.util.List;
 
@Path("/backend")
public class UserResource {
 
    private UserDAO userDAO;
 
    public UserResource(UserDAO userDAO) {
        super();
        this.userDAO = userDAO;
    }
 
    @GET
    @Path("/user")
    @Consumes({MediaType.APPLICATION_JSON})
    @Produces({MediaType.APPLICATION_JSON})
    public List<User> fetch(){
 
        return userDAO.findAll();
    }
 
    @PUT
    @Path("/user/{id}")
    @Consumes({MediaType.APPLICATION_JSON})
    @Produces({MediaType.APPLICATION_JSON})
    public void update(@PathParam("id") Long id, User user)
    {
        userDAO.update(
                id,
                user.getFirstName(),
                user.getLastName(),
                user.getEmail()
        );
    }
 
    @POST
    @Path("/user")
    @Consumes({MediaType.APPLICATION_JSON})
    @Produces({MediaType.APPLICATION_JSON})
    public void add(User user)
    {
        userDAO.insert(user.getId(), user.getFirstName(), user.getLastName(), user.getEmail());
    }
 
    @DELETE
    @Path("/user/{id}")
    @Consumes({MediaType.APPLICATION_JSON, MediaType.APPLICATION_XML})
    @Produces({MediaType.APPLICATION_JSON})
    public void delete(@PathParam("id") Long id)
    {
        userDAO.delete(id);
    }
 
}

I actually implemented the ability to add/remove/list/update first in Backbone and it was a lot of help to use Christophe Coenraet’s Wine Application here to help make sure I had a good working example. If you want to take a look at my example application running with Backbone it is located here.

After finishing the Backbone I was able to implement the same exact approach in Angular to see how it sized up to the Backbone approach. Overall this has been a good exercise but I am interested to put it out there and try to solicit some feedback and help nail down which of these is the best one to move forward with for any project. To take a look at the Angular example just have a look here.

A few things I noticed about the two frameworks

Backbone appeared to have a reference to the DOM and heavily relied on updating the contents of the DOM from the JS. Conversely Angular appears to use more of an MVVP and also had features for binding the view to a model object. The overall size of the Javascript required to accomplish the exact same goal was smaller with Angular vs Backbone by nearly 1/3rd. Number of lines of code doesn’t matter though, as readability is important (just look at perl programs, some of those can be done in a few lines and the equivalent in Java might be 100s of lines).

Technically speaking the Rest hooks from Backbone were very strong, and they work as documented. The Angular Rest hooks took quite a bit of adjusting and were very poorly documented. You can have a look at Angular’s $resource here but you will notice that the documentation lacks some direction and the examples actually didn’t work for me. I think that this will improve in time, as the site took a drastic change in the past few weeks.

I look forward to exchanging some opinions on this as I want to take a fair look at the available options. If you have anything to add or want to get the sample running try to take a look at the readme I posted on GitHub. If you have any issues let me know and I will work to iron it out either via comment or by updating the GitHub code.

In the process of setting up my new server environment I have learned quite a few new little tricks to make this work. Firstly lets walk through the environment as I planned it before implementation

The key to note here is that I only have one externally facing IP address. I then have a decent router capable of port forwarding (any home model router will do here) at the front of my network. It routes traffic to the QNAP NAS and the Server running the virtual machines. You can also see that I setup a SAN between the QNAP and Server on the 10GBe network connections. The IP Addresses are not actually correct anymore but in terms of planning this worked out to help me wrap my head around what I wanted to do.

The DomO (server) is running CentOS 5.X since it had best hardware support – reasonable end of life and is generally secure and stable at this point. All of my DomU instances are running CentOS 6.X because they do the real work and I want the newest packages to install on them.  Most of my instances are used for serving some sort of web content or application server content. All of them are proxied behind a dedicated Apache instance running ProxyPass (mod_proxy) rules to connect either via AJP or HTTP to the other instances. This allows my router to just forward all traffic coming in on port 80 to the dedicated Apache instance. I may later evaluate things like HAProxy or just some load balanced Apache solution, but for now this works as I am not dealing with high throughput traffic – maybe one day that’ll be a great problem to have and I will just move this setup to some fatter pipes.

It is worth noting that CentOS has some SE Linux rules enabled by default and they will break ProxyPass without making these changes:

http://sysadminsjourney.com/content/2010/02/01/apache-modproxy-error-13permission-denied-error-rhel

Immediately allow apache to connect out to other ports

/usr/sbin/setsebool httpd_can_network_connect 1

Permanently allow apache to connect out to other ports

/usr/sbin/setsebool -P httpd_can_network_connect 1

My DomO instance is synchronizing the entire CentOS distribution down via RSync so that installing new packages is super fast and occurs over the 1.x.x.x network (10GBe) rather than fetching from the outside world.

This is how I accomplished that

0 */6 * * * /usr/bin/rsync -avvzH –delete rsync://mirror.anl.gov/centos/ /opt/ISOS/CentOS/ > /tmp/centos.rsync.log

I noticed that I needed to be able to connect up to my machine remotely to adminster it so I installed X11VNC on the Dom0 instance and invoke it by calling these commands via SSH from anywhere I want to connect. I will likely lock this down to only accept ssh connections from certain IPs:

Connect to server with the following command and launch x11vnc

ssh -L 5900:localhost:5900 root@host.com ‘x11vnc’

or if no one is logged in yet

ssh -L 5900:localhost:5900 root@host.com ‘x11vnc -auth /var/gdm/\:0.Xauth’

After this I can run the command:

vncviewer localhost:0

(Reference above http://wiki.zenwalk.org/index.php?title=HOWTO_Use_x11vnc)

to connect to my machine and see the desktop. This is useful if I need to use graphical installers to configure my other DomU instances. This was a blocker to install Glassfish actually for my NETS instances so now it is good that I have solved that issue.

With respect to actually spinning up new servers, I installed a virtual machine by following documentation I found here:

http://pbraun.nethence.com/doc/sysutils_xen/dom0.html

However after creating a vm-skeleton and leaving it turned off I was able to clone it via a script to get me a few machines installed quickly and allocated.

Example of creating a new server:

virt-install -p –name=vm-1 –location=/opt/ISOS/CentOS/6/isos/x86_64/CentOS-6.2-x86_64-bin-DVD1.iso –bridge=xenbr0 –ram=4096 –disk path=/var/lib/xen/images/vm-skeleton,size=40,sparse=false

Duplication of servers (creates 10 servers)

for i in 1 2 3 4 5 6 7 8 9 10 ; do virt-clone -o vm-skeleton -n vm-$i -f /var/lib/xen/images/vm-$i ; done

The last issue I had to resolve was related to using AJP with Glassfish. I was creating proxypass rules like this:

<VirtualHost *:80>

ProxyPass / ajp://192.168.1.61:8080/ keepalive=on timeout=600

ProxyPassReverse / ajp://192.168.1.61:8080/

ServerName nets.neosavvy.com

</VirtualHost>

However the default number of threads that Apache allocates is 100 but the default that Glassfish will listen for is 5. So quickly Apache will overload Glassfish and start queuing requests and will cause them to timeout. Errors look like this:

[Wed Jan 18 14:08:59 2012] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can’t receive header

[Wed Jan 18 14:08:59 2012] [error] ajp_read_header: ajp_ilink_receive failed

[Wed Jan 18 14:08:59 2012] [error] (120006)APR does not understand this error code: proxy: read response failed from (null) (192.168.1.61)

[Wed Jan 18 14:09:33 2012] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can’t receive header

[Wed Jan 18 14:09:33 2012] [error] ajp_read_header: ajp_ilink_receive failed

[Wed Jan 18 14:09:33 2012] [error] (120006)APR does not understand this error code: proxy: read response failed from (null) (192.168.1.61)

All I had to do to fix this was to change the http-thread-listener’s thread-pool to be 100 instead of 5.

Reference (http://www.java.net/forum/topic/glassfish/glassfish/jkenabled-thread-leak)

I am sure I will have more notes to post and hopefully these will help someone out when they are setting up their own small network.

Recently I have been working on an event for programmers and business folks here in NYC. It is quite the task and requires a lot of work and the initial cost of having an event here is actually worth writing about.

There are a few things you need to have a good event here or anywhere for that matter

1. Space to adequately and comfortably seat ~75 people
2. Food and Beverages for them to eat and be comfortable during the event
3. Prizes to give away to the winners
4. General supplies (pens/paper etc)

So we have found some great space to have an event but ultimately it ranges drastically from $1000 per day to $7000 per day. This immediately increases the cost of having an event to something that would require some corporate sponsorship (at the 7k side – not the 1k side).

Then of course there is the food, we wanted to have some great beer so a friend of ours is brewing us a batch of good beer. Kegs of beer would have cost us $500 or so for ~75 people. Just think back to your fraternity days to draw on knowledge of price/keg of good beer. Not natural light – the good stuff like Sierra Nevada. He is charging us ~$800 and includes a pint glass for every person who shows up with the event logo on it. [ http://www.brewheister.com/ ]

Food is a tough one but feeding ~75 people pizza you are looking at spending about 600 and then if you want to provide some snacks you are probably going to spend upwards of $1000.

Overall breakdown of the budget to have an event in NY is at minimum about $5000 when you factor in the cost of some prizes. So having a 50 person turnout would require a $100 ticket price just to break even. Now keep in mind I am not trying to put together an event to make a profit, I want to throw a kickass event for the tech community in NY. No one is going to be willing to pay $100 for a ticket for a one day event though, but I am wondering if a $70 price tag is legit and a pricepoint people are willing to pay.

Thoughts anyone?

My primary client for most of the last 8 months has been located in NYC and given Neosavvy is a NC Incorporated business I have learned quite a bit about traveling to NY for business.

Firstly you must be registered as a Foreign Corporation to do work when physically located there even if just for one day. They are the strictest state I have read about in regards to work location and paying taxes so this as it turns out is really important. Not only must you be a registered corporation but you must also elect the Subchapter S accounting status with NY if you are actually managing your accounting that way. Most states in the Union will just let you observe your federal tax election, however NY in its ever aiding willingness to complicate business makes you file it again with them. If you are in my situation where you don’t make a timely election (within 3 months) you will have to file a plea to request they let you make a late election. Mine is pending, and right now Neosavvy is being treated like a C Corporation which unfortunately is causing some really unexpected double taxation which hopefully others can avoid by reading this.

Federal Sub Chapter S Election form
Election by Federal S Corporation to be treated as a New York S Corporation

The second thing is that if you employ anyone even for 1 hour a month and they physically are doing work in NY even just for one business meeting, you are required to have a workers compensation policy. Any decent sized company should probably have workers compensation, but NY must be explicitly mentioned in the policy as a insured location. This can cause some really hefty fines if you don’t comply as the law says it’s a $2000 / 10 Day period of non-compliance. They will let the fine accrue to quite a high amount before they notify you as well. So be sure to set up a workers compensation policy, its cheap $500/year should cover a small team of software developers or accountants, or other pencil pusher-esque jobs.

I am currently using the PayCycle pay as you go workers compensation policy. If you use PayCycle for payroll (which is very cost effective vs PayChex and ADP) you can have them take care of disability insurance and workers compensation. They are an Intuit Company and are really helpful for simplifying the process for those of us small enough to be using QuickBooks.

There are some other travel related accounting principles worth noting as well that are specific to NY, I am unsure of how lenient they are on it at this time. Primarily regarding long term travel to NY. After a period of 12 months you are considered a resident and can no longer expense travel to NY. I question this to some degree because I know of consultants doing a lot of work in NY traveling here for very long periods, easily over 3 years, and they are expensing their corporate apartments and regular meal per diems long beyond the 12 month period my CPA has instructed me to observe. So it seems even if you have negotiated travel terms for your work you will be unable to show those as non-taxable expense for a period of 12 months. This could be incorrect but definitely worth bringing up with your CPA when beginning any work in NYC.

I hope this helps people please email me or comment if you have similar experiences as I am working my way through understanding how to be a upstanding business citizen of NYC.