User Defined Functions in Hive
By dan.mcclary on Apr 02, 2013
User-defined Functions (UDFs) have a long history of usefulness in SQL-derived languages. While query languages can be rich in their expressiveness, there's just no way they can anticipate all the things a developer wants to do. Thus, the custom UDF has become commonplace in our data manipulation toolbox.
Apache Hive is no different in this respect from other SQL-like languages. Hive allows extensibility via both Hadoop Streaming and compiled Java. However, largely because of the underlying MapReduce paradigm, all Hive UDFs are not created equally. Some UDFs are intended for "map-side" execution, while others are portable and can be run on the "reduce-side." Moreover, UDF behavior via streaming requires that queries be formatted so as to direct script execution where we desire it.
The intricacies of where and how a UDF executes may seem like minutiae, but we would be disappointed time spent coding a cumulative sum UDF only executed on single rows. To that end, I'm going to spend the rest of the week diving into the three primary types of Java-based UDFs in Hive. You can find all of the sample code discussed here.
The Three Little UDFs
Hive provides three classes of UDFs that most users are interested in: UDFs, UDTFs, and UDAFs. Broken down simply, the three classes can be explained as such:
- UDFs -- User Defined Functions; these operate row-wise, generally during map execution. They're the simplest UDFs to write, but constrained in their functionality.
- UDTFs -- User Defined Table-Generating Functions; these also execute row-wise, but they produce multiple rows of output (i.e., they generate a table). The most common example of this is Hive's explode function.
- UDAFs -- User Defined Aggregating Functions; these can execute on either the map-side or the reduce-side and far more flexible than UDFs. The challenge, however, is that in writing UDAFs we have to think not just about what to do with a single row, or even a group of rows. Here, one has to consider partial aggregation and serialization between map and reduce proceses.