Thursday, November 16, 2017

Create Dataframe and play with it

We may want to create a spark data frame on the fly given a Json String. Below is a code sample that can run on spark shell.

val jsonStr = """{"startDate":"2017:12:30"}"""
val rdd = sc.parallelize(Seq(jsonStr))
var df = sqlContext.read.json(rdd)
df.registerTempTable("temp")
val sqlResult = sqlContext.sql("select * from temp")
sqlResult.show()

output:

+———-+
| startDate|
+———-+
|2017:12:30|
+———-+

now say i update the original dataframe

df = sqlResult.map(row=>{"2012:09:09"}).toDF
df.show

output

+———-+
| _1|
+———-+
|2012:09:09|
+———-+

in the above if you see we are replacing the original dataframe. –meaning if in the tasks we are referencing the df again and get the result.. we will be getting the new result. So can this be used as poormans cache? I am yet to discover the issues!

Print the dataframe::
Lets say that we want to print every entry of the dataframe…

df.foreach( r => print(r.getValuesMap(r.schema.fieldNames)))

which gives output as below

Map(startDate -> 2017:12:30)

Written with StackEdit.

Tuesday, July 11, 2017

Best Tabbed SSH terminal/ tabbed Putty

My first choice was Putty-Connection manager. The problem is it suffers weird issues. Even if u solve all of them you endup with Alt + Tab issue. I.e. you hit on Alt tab and still stay on the putty connection manager.

Then i tried MTPutty. The UI is very good. But it suffers the same Alt Tab issue of Putty connection manager

Then Poderosa/ CMDER. They dont suffer Alt tab issue but they cant remember the sessions either. Again waste.

Couple of days back i came across nd-putty. The only problem with it was the font rendering issue. The fonts render blurred. I contacted the deverloper and finally narrowed down that its Windows 10 issue. When you put magnification of display to 125% (right click > display etc). Finally i got this resolved. its basically DPI issue. Right click on the putty-nd; in properties click on the tab: compatibility > Disable display scaling on high DPI settings. And this did the magic.

Now i swithced to PuttyND.

more info: http://windows10_dpi_blurry_fix.xpexplorer.com/

Friday, November 11, 2016

Q

I recall something Dr. Martin Luther King, Jr. said 50 years ago: "If you can't fly, then run. If you can't run, then walk. If you can't walk, then crawl, but whatever you do you have to keep moving forward." This advice is timeless