All posts by admin

JavaCard Applets






STK Applets – simhacks

JavaCard Applets

Writing, compiling, loading, and using JavaCard STK SIM Applets.

Java Card refers to a software technology that allows Java-based applications (applets) to be run securely on smart cards and similar small memory footprint devices. Java Card is the tiniest of Java platforms targeted for embedded devices. Java Card gives the user the ability to program the devices and make them application specific. It is widely used in SIM cards (used in GSM mobile phones) and ATM cards.

wikipedia

Eclipse

Remove the JRE system library and add the Java Card library. Set compiler level to 1.3 and bytecode compatibility to 1.1

Oops, this will be here shortly!

Shadysim is a collection of Makefiles and python scripts that makes it easy to compile and load SIM applets. See https://github.com/shadytel/sim-tools
for more information.





Smart Card Standards

simhacks

DEFCON 21 Talk
SIM Cards
JavaCard STK Applets
Android Emulator
Resources
Store

SIM Cards

Random notes about interfacing with SIM cards.

Smart Card Standards
GlobalPlatform
References

Smart Card Standards

ISO 7816-1: Physical characteristics
ISO 7816-2: Electrical contacts
ISO 7816-3: Electrical interface, Transmission Protocol Data Units (TPDUs)
T=0: Byte-oriented protocol
T=1: Block-oriented protocol
ISO 7816-4: Standard commands, Application Protocol Data Units (APDUs)
ISO 14443-4: “T=CL”: APDUs over RFID

GlobalPlatform
At install time, you can specify:

Initialization data
App parameters
STK uses these extensively
Privileges
Priority
Number of menu items
Max menu item size

Other fun commands:

List AIDs, including both modules and instances
Delete AIDs
You MUST delete instances before deleting the executable!
You MUST delete old AIDs before reusing them!

References

Java Card 2.1.1 Virtual Machine Specification
GlobalPlatform card specification 2.1/2.2
GSM 03.48 – Secure remote SIM access
GSM 03.40 – SMS standard
ETSI TS 101 220 – Assigned numbers
ETSI TS 102 221 – UICC/(U)SIM spec
ETSI TS 102 223 – Card Application Toolkit
ETSI TS 102 226 – Remote APDUs
ETSI TS 102 241 – UICC/SIM API for JavaCard

Tim Berners-Lee tells us his radical new plan to upend the World Wide Web With an ambitious decentralized platform, the father of the web hopes it’s game on for corporate tech giants like Facebook and Google.

09.29.18

Exclusive: Tim Berners-Lee tells us his radical new plan to upend the World Wide Web
With an ambitious decentralized platform, the father of the web hopes it’s game on for corporate tech giants like Facebook and Google.

By Katrina Brooker5 minute Read

Last week, Tim Berners-Lee, inventor of the World Wide Web, asked me to come and see a project he has been working on almost as long as the web itself. It’s a crisp autumn day in Boston, where Berners-Lee works out of an office above a boxing gym. After politely offering me a cup of coffee, he leads us into a sparse conference room. At one end of a long table is a battered laptop covered with stickers. Here, on this computer, he is working on a plan to radically alter how all of us live and work on the web.

“The intent is world domination,” Berners-Lee says with a wry smile. The British-born scientist is known for his dry sense of humor. But in this case, he is not joking.

This week, Berners-Lee will launch Inrupt, a startup that he has been building, in stealth mode, for the past nine months. Backed by Glasswing Ventures, its mission is to turbocharge a broader movement afoot, among developers around the world, to decentralize the web and take back power from the forces that have profited from centralizing it. In other words, it’s game on for Facebook, Google, Amazon. For years now, Berners-Lee and other internet activists have been dreaming of a digital utopia where individuals control their own data and the internet remains free and open. But for Berners-Lee, the time for dreaming is over.

“We have to do it now,” he says, displaying an intensity and urgency that is uncharacteristic for this soft-spoken academic. “It’s a historical moment.” Ever since revelations emerged that Facebook had allowed people’s data to be misused by political operatives, Berners-Lee has felt an imperative to get this digital idyll into the real world. In a post published this weekend, Berners-Lee explains that he is taking a sabbatical from MIT to work full time on Inrupt. The company will be the first major commercial venture built off of Solid, a decentralized web platform he and others at MIT have spent years building.
A Netscape for today’s internet

If all goes as planned, Inrupt will be to Solid what Netscape once was for many first-time users of the web: an easy way in. And like with Netscape, Berners-Lee hopes Inrupt will be just the first of many companies to emerge from Solid.

“I have been imagining this for a very long time,” says Berners-Lee. He opens up his laptop and starts tapping at his keyboard. Watching the inventor of the web work at his computer feels like what it might have been like to watch Beethoven compose a symphony: It’s riveting but hard to fully grasp. “We are in the Solid world now,” he says, his eyes lit up with excitement. He pushes the laptop toward me so I too can see.

On his screen, there is a simple-looking web page with tabs across the top: Tim’s to-do list, his calendar, chats, address book. He built this app–one of the first on Solid–for his personal use. It is simple, spare. In fact, it’s so plain that, at first glance, it’s hard to see its significance. But to Berners-Lee, this is where the revolution begins. The app, using Solid’s decentralized technology, allows Berners-Lee to access all of his data seamlessly–his calendar, his music library, videos, chat, research. It’s like a mashup of Google Drive, Microsoft Outlook, Slack, Spotify, and WhatsApp.

The difference here is that, on Solid, all the information is under his control. Every bit of data he creates or adds on Solid exists within a Solid pod–which is an acronym for personal online data store. These pods are what give Solid users control over their applications and information on the web. Anyone using the platform will get a Solid identity and Solid pod. This is how people, Berners-Lee says, will take back the power of the web from corporations.

[Image courtesy of Tim Berners-Lee]
For example, one idea Berners-Lee is currently working on is a way to create a decentralized version of Alexa, Amazon’s increasingly ubiquitous digital assistant. He calls it Charlie. Unlike with Alexa, on Charlie people would own all their data. That means they could trust Charlie with, for example, health records, children’s school events, or financial records. That is the kind of machine Berners-Lee hopes will spring up all over Solid to flip the power dynamics of the web from corporation to individuals.

A new revolution for developers?

Berners-Lee believes Solid will resonate with the global community of developers, hackers, and internet activists who bristle over corporate and government control of the web. “Developers have always had a certain amount of revolutionary spirit,” he observes. Circumventing government spies or corporate overlords may be the initial lure of Solid, but the bigger draw will be something even more appealing to hackers: freedom. In the centralized web, data is kept in silos–controlled by the companies that build them, like Facebook and Google. In the decentralized web, there are no silos.

Starting this week, developers around the world will be able to start building their own decentralized apps with tools through the Inrupt site. Berners-Lee will spend this fall crisscrossing the globe, giving tutorials and presentations to developers about Solid and Inrupt. (There will be a Solid tutorial at our Fast Company Innovation Festival on October 23.)

“What’s great about having a startup versus a research group is things get done,” he says. These days, instead of heading into his lab at MIT, Berners-Lee comes to the Inrupt offices, which are currently based out of Janeiro Digital, a company he has contracted to help work on Inrupt. For now, the company consists of Berners-Lee; his partner John Bruce, who built Resilient, a security platform bought by IBM; a handful of on-staff developers contracted to work on the project; and a community of volunteer coders.

Later this fall, Berners-Lee plans to start looking for more venture funding and grow his team. The aim, for now, is not to make billions of dollars. The man who gave the web away for free has never been motivated by money. Still, his plans could impact billion-dollar business models that profit off of control over data. It’s not likely that the big powers of the web will give up control without a fight.

When asked about this, Berners-Lee says flatly: “We are not talking to Facebook and Google about whether or not to introduce a complete change where all their business models are completely upended overnight. We are not asking their permission.”

After Years of Abusive E-mails, the Creator of Linux Steps Aside

After Years of Abusive E-mails, the Creator of Linux Steps Aside

By Noam Cohen

September 19, 2018

After years of verbally abusing programmers who contribute to the Linux operating-system kernel he created, the celebrated coder Linus Torvalds is stepping aside and says he is getting help.
Photograph by Kimmo Mäntylä / REX / Shutterstock

The e-mails of the celebrated programmer Linus Torvalds land like thunderbolts from on high onto public lists, full of invective, insults, and demeaning language. “Please just kill yourself now. The world will be a better place,” he wrote in one. “Guys, this is not a dick-sucking contest,” he observed in another. “SHUT THE FUCK UP!” he began in a third.

Torvalds has publicly posted thousands of scathing messages targeting programmers who submit what he deems flawed code to the Linux computer-operating-system kernel, which he brought to life more than twenty-five years ago and now administers as a collaborative, open-source project. Today, the Linux kernel is famous, running the enormous computers of Google, PayPal, Amazon, and eBay, and the two billion mobile phones using the Android operating system. Torvalds, though, retains final say over each precious line of code, just as he did when he first started working on the system as a graduate student at the University of Helsinki. For years, he has been known as Linux’s “benevolent dictator for life.”

On Sunday, the benevolent dictator announced that he would be stepping down temporarily, to “get some assistance on how to understand people’s emotions and respond appropriately.” Torvalds, who is forty-eight and lives with his family outside Portland, Oregon, made clear that he wasn’t burned out. “I very much do want to continue to do this project that I’ve been working on for almost three decades,” he wrote in a post to the Linux-kernel mailing list. “I need to take a break to get help on how to behave differently and fix some issues in my tooling and workflow.” Torvalds named a deputy, Gregory Kroah-Hartman, to run the project while he was away.

Torvalds’s decision to step aside came after The New Yorker asked him a series of questions about his conduct for a story on complaints about his abusive behavior discouraging women from working as Linux-kernel programmers. In a response to The New Yorker, Torvalds said, “I am very proud of the Linux code that I invented and the impact it has had on the world. I am not, however, always proud of my inability to communicate well with others—this is a lifelong struggle for me. To anyone whose feelings I have hurt, I am deeply sorry.”

Torvalds’s response was conveyed by the Linux Foundation, which supports Linux and other open-source programming projects and paid Torvalds $1.6 million in annual compensation as of 2016. The foundation said that it supported his decision and has encouraged women to participate but that it has little control over how Torvalds runs the coding process. “We are able to have varying degrees of impact on these outcomes in newer projects,” the statement said. “Older more established efforts like the Linux kernel are much more challenging to influence.”

Until this weekend, Torvalds had not only defended his aggressive behavior but insisted that it contributed to Linux’s runaway success. “If you want me to ‘act professional,’ I can tell you that I’m not interested,” he wrote in 2013, in response to a prominent Linux contributor, Sage Sharp, who demanded on a public e-mail list that Torvalds stop using “physical intimidation, verbal threats or verbal abuse” in his e-mails. “I’m sitting in my home office wearign [sic] a bathrobe,” Torvalds wrote. “The same way I’m not going to start wearing ties, I’m also not going to buy into the fake politeness, the lying, the office politics and backstabbing, the passive aggressiveness, and the buzzwords. Because THAT is what ‘acting professionally’ results in: people resort to all kinds of really nasty things because they are forced to act out their normal urges in unnatural ways.”

Although it distributes its product for free, the Linux project has grown to resemble a blue-chip tech company. Nominally a volunteer enterprise, like Wikipedia, Linux, in fact, is primarily sustained by funds and programmers from the world’s large technology companies. Intel, Google, IBM, Samsung, and other companies assign programmers to help improve the code. Of the eighty thousand fixes and improvements to Linux made in the past year, more than ninety per cent were produced by paid programmers, the foundation reported in 2017; Intel employees alone were responsible for thirteen per cent of them. These same companies, and hundreds of others, covered the foundation’s roughly fifty-million-dollar annual budget.

Linux’s élite developers, who are overwhelmingly male, tend to share their leader’s aggressive self-confidence. There are very few women among the most prolific contributors, though the foundation and researchers estimate that roughly ten per cent of all Linux coders are women. “Everyone in tech knows about it, but Linus gets a pass,” Megan Squire, a computer-science professor at Elon University, told me, referring to Torvalds’s abusive behavior. “He’s built up this cult of personality, this cult of importance.”
Video From The New Yorker
How Fortnite Captured Teen-age Minds

For a research project, Squire used e-mails from Torvalds to train a computer to recognize insults. According to Squire’s tabulations, more than a thousand of the twenty-one thousand e-mails Torvalds sent in a four-year period used the word “crap.” “Slut,” “bitch,” and “bastard” were employed much less frequently during that period. Squire told me that she found few examples of gender bias. “He is an equal-opportunity abuser,” she said. Squire added, though, that for non-male programmers the hostility and public humiliation is more isolating. Over time, many women programmers leave the community. “Women throw in the towel first,” she told me. “They say, ‘Why do I need to put up with this?’ ”

Linus Torvalds and Linux Code of Conduct: 7 myths debunked

Linus Torvalds and Linux Code of Conduct: 7 myths debunked

No, protesting programmers are not removing code from Linux; there are no purges of politically incorrect Linux kernel developers. And Linus Torvalds is coming back.
Steven J. Vaughan-Nichols

By Steven J. Vaughan-Nichols for Linux and Open Source | September 25, 2018 — 16:04 GMT (09:04 PDT) | Topic: Enterprise Software

Since Linus Torvalds announced he was taking time off to work on his behavior in the Linux developer community and a new Linux kernel developer Code of Conduct (CoC) was introduced, there has been endless malarkey written about both moves.

Here are some of the more hysterical myths regarding what’s happening:

Myth 1. The Linux kernel has been taken over by Social Justice Warriors (SJW)!

Hardly! The new CoC is based on Coraline Ada Ehmke’s Contributor Covenant, version 1.4. It’s been been adopted by many open-source projects such as Eclipse, Ruby, and Kubernetes.

While the code isn’t that controversial, Ehmke has spoken out against transphobia, which has been seen by some as pushing her own agenda on open-source projects. Her sarcastic tweet (“I can’t wait for the mass exodus from Linux now that it’s been infiltrated by SJWs. Hahahah.”) helped bring on the flames. But Ehmke has no leadership role in the Linux community. And Ehmke later tweeted, “Maybe they think I have a commit bit on the Linux kernel? Maybe they can’t read git and so they think that I merged the commit. ”

Also: Tech industry is leaving behind women of color, report shows CNET

Torvalds, with other senior Linux kernel developers, added the CoC. No one else did. The Linux Foundation’s Linux Technical Advisory Board (TAB) is in charge of enforcing it. Theodore “Ted” T’so, a senior Linux kernel developer and Google engineer, explained on the LKML, “The TAB can make a recommendation, but the decision to act on that recommendation resides with the Maintainers in general, and ultimately, Linus.”

Sage Sharp, the former kernel developer who left the Linux community because of its toxicity, doesn’t trust TAB to do its duty by the code. Sharp tweeted, “I have no faith that the Linux Foundation Technical Advisory Board will respond to a Code of Conduct violation promptly or with a well-thought out response. Please push the board to release an anonymized transparency report on all past Linux kernel Code of Conduct violation cases.”

Does this sound like SJWs are taking over Linux? I don’t think so.

Myth 2. Linus Torvalds left and then adopted the CoC to get ahead of The New Yorker article about his discouraging women from working on the kernel.

It played a factor. But, having known Torvalds for almost 30 years, I doubt it mattered much to him. There was no new news in the article. The Linux Kernel Mailing List (LKML) is open, and Torvald’s sometimes angry management style has been on display for decades. As Torvalds wrote, he realized he “had been ignoring some fairly deep-seated feelings in the community.”

Also: ‘There’s a culture that works against women’

Torvalds added, “I am not an emotionally empathetic kind of person and that probably doesn’t come as a big surprise to anybody. Least of all me. The fact that I then misread people and don’t realize (for years) how badly I’ve judged a situation and contributed to an unprofessional environment is not good.”

Myth 3. Nothing is going to change.

Personally, I think the Code of Conduct in and of itself won’t make much of a difference. I believe you can’t legislate morality. What Torvalds does when he gets back will make the difference. He has always set the Linux kernel community’s tone, and he will again.

Myth 4. Linus won’t/can’t change.

I think he will change. In any case, he should be given a chance.

As VM (Vicky) Brasseur, open-source consultant and VP of the Open Source Initiative (OSI), tweeted, “Folks, I know there’s a lot of work yet to be done and that this is only the first step, but if you criticise someone for taking that first step (even if you think they should’ve done it much earlier), there’s less likely to be more steps after that.”

Also: The state of women in computer science TechRepublic

Myth 5. Purges of Linux kernel developers have begun.

No. And, there’s no sign of this happening.

Myth 6. Developers are leaving Linux and taking their code with them.

No, they’re not leaving.

Yes, someone under the nom de plume “unconditionalwitness,” who had never posted to the LKML before under that name, wrote that people ejected from the Linux Kernel Community due to the code could “rescind the license grant regarding their property via written notice to those whom they are rescinding the grant from (regarding their property (code).”

In other words, they can remove their code from Linux. But no one has been ejected from the Linux kernel community, and no one has removed their code. To the best of my knowledge, no one has filed a complaint against anyone to the TAB. Even if someone was kicked out, it’s unclear what would happen to the code they had already donated.

Eric S. Raymond, one of the creators of the open-source concept, wrote, “This threat has teeth. I researched the relevant law when I was founding the Open Source Initiative. In the US there is case law confirming that reputational losses relating to conversion of the rights of a contributor to a GPLed project are judicable in law.”

Also: Women are better in tech than men, says a report CNET

Richard M. Stallman, who wrote the GPLv2 and champions free software, is purported to have written, “The developers of Linux, or any free program, can remove any and all code, at any time, without giving a reason. However, this doesn’t force others to delete that code from their own versions of the program.”

Heather Meeker, a partner at the law firm O’Melveny & Myers, who specializes in open-source software licensing, wrote, “Copyright ownership in large projects such as the Linux kernel is complicated. It’s like a patchwork quilt. When developers contribute to the kernel, they don’t sign any contribution agreement or assignment of copyright. The GPL covers their contributions, and the recipient of a copy of the software gets a license, under GPL, directly from all the authors. (The kernel project uses a document called a Developer Certificate of Origin, which does not grant any copyright license.) The contributors’ individual rights exist side-by-side with rights in the project as a whole.”

Also: Diversity at Google hasn’t changed much over the last year CNET

So, what’s what? No lawyer has spoken directly to the point on this issue, and it’s never come up in court. I am inclined to doubt that the code can be removed, or if it could, that it would have any practical effect on the kernel.

As Matthew Garrett, Linux developer and Google security engineer, remarked on Twitter, “For every person that leaves the kernel over the CoC, I pledge to help mentor a new contributor interested in taking over their work.” He quickly followed up, “Well, so far I’ve got like an order of magnitude more people interested in contributing to the kernel than people who have credibly threatened to stop contributing, so that seems like a good sign.”
Featured stories

Microsoft and Salesforce race to offer a single view of customers’ data
Will there be an October Apple event? Signs point to yes
How to decide: Picking the 2018 iPhone that’s right for you
Kardia EKG after Apple Watch: AliveCor’s future of remote patient care

Myth 7. Linus is never coming back!

Linus is coming back.

Greg Kroah-Hartman, Linux kernel maintainer and pro tempore head of Linux, signed the Linux 4.19-rc5 release message, “Greg ‘keeping the seat warm for a few weeks’ k-h.”

He will return when he feels like it.

If I were a betting man, I’d say he’ll be back in time to manage the next release cycle, 4.20/5.0, later this fall. Or, to be terribly precise, Oct. 22, 2018, when Linux Kernel Maintainer Summit will be held in Edinburgh Scotland. After all, one of the reasons Torvalds stepped back from the kernel was because this event had to be rescheduled to meet his schedule.

Also: Linus Torvalds talks frankly about Intel security bugs

When all is said and done — for all the hubbub about the CoC and Torvalds stepping out for a while — Linux is still being developed. And, none of the worst case scenarios has shown any signs of happening. For all the sturm und drang, Linux continues on.

One group dominates the second round of net neutrality comments

by Andrew Pendleton and Bob Lannon

Dec. 16, 2014, 1:23 p.m.

A letter-writing campaign that appears to have been organized by a shadowy organization with ties to the Koch Brothers inundated the Federal Communications Commission with missives opposed to net neutrality (NN), an analysis by the Sunlight Foundation reveals.

Over the past several months, the Federal Communications Commission has been working towards a new set of rules around net neutrality, and a large part of that process has been accepting comments from the public. In September, we reported on our analysis of the comments from the first comment period of this rulemaking, and we’d now like to take a look at the comments from the second, which the FCC released in bulk in October. We again used natural language processing techniques to examine the approximately 1.6 million comments we successfully extracted from this batch of comments, helping to expose important topics discussed in the comments, and to group similar comments together.

Among our key findings from round two:

In marked contrast to the first round, anti-net neutrality commenters mobilized in force for this round, and comprised the majority of overall comments submitted, at 60%. We attribute this shift almost entirely to the form-letter initiatives of a single organization, American Commitment, who are single-handedly responsible for 56.5% of the comments in this round.

Who’s behind the group that flooded the FCC with anti-net neutrality comments?
American Commitment, the group behind a majority of the recent anti-net neutrality comments, is affiliated with the Koch brothers’ network. Read more.

In large part because of this campaign, the percentage of comments submitted that we believe to have been form letter submissions was significantly higher for this round than the last one, at 88%.

Non-form-letter submissions had a similar sentiment distribution as comments in the first round, at less than 1% opposed to net neutrality.

In general, many more comments were difficult to classify in this round than in the first round. Some of the new campaigns on the anti-net neutrality side appear to have been crafted to use similar language to the successful pro-neutrality campaigns of the first round, while supporting opposite conclusions, and many non-form-letter comments used talking points from both camps, making their ultimate intents unclear.

As with the last round, the corpus also included submissions on behalf of telecommunications firms, advocacy organizations, etc., which were written using formal legal language that set them apart from the bulk of the comments. Again, these were a tiny fraction of a percent of overall comments.

Combined with the first round comments, we characterize 41% of the total comments submitted as being anti-net neutrality (with the balance being a mix of pro-NN and comments with no clear opinion), and we estimate that 79% of submissions came as part of form letter campaigns.

Below is a revised version of our comment visualization tool, this time exploring the data from the second comment period.
Graphic credit: Sunlight Foundation. To embed this on your site, click “embed this widget.”

We again did a deep dive into the topics that came to light from this model. As expected many of the same topics recurred in the comments in this round:

Opposition to paid priority or tiered speed was again commonly discussed in pro-NN comments. Form letter campaigns discussing this topic included those from FreePress, BattleForTheNet, Credo, Daily Kos and the Sierra Club.

Many commenters again discussed various legal rationales for net neutrality, with phrases like “common carrier,” “title II,” and “public utility.” Such phrases occurred in about half of comments in this iteration.

Arguments about the economy were common in both pro-NN and anti-NN comments, with disagreements as to which policy best favored economic growth.

Additionally, particularly on the now-better-represented anti-net neutrality side, some new framings were apparent:

Similar to, but less ambiguous than, the messaging that emerged from tea-partier groups in round one, was a set of arguments that dominated the anti-NN comments in round two, and that we believe originated with conservative activist organization American Commitment. Comments from this campaign had a shared template, with different targeted messages inserted between the second and third paragraph. Those targeted messages centered on topics as far ranging as personal freedoms, economic threats, the poor state of US public utilities, and the characterization of pro-NN advocates as extreme leftists (Free Press’s Robert McChesney is portrayed as a Communist).

A separate, smaller contingent that opposed FCC action on net neutrality suggested that while net neutrality regulation might be within the government’s purview, it would be better left to Congress. Most of the comments in this group came from a form letter campaign organized by TechFreedom.

Our identification of form letters followed the same approach as last round: identify clusters with particularly low variance and peruse them to confirm shared boilerplate language. This task was much easier with the second round, however, because there was less noise within each cluster. Because the corpus as a whole contained mostly form letters, partitioning it into clean “neighborhoods” was not difficult. Also, the uniformity of the comments submitted through campaigns like American Commitment’s, TechFreedom’s and BattleForTheNet’s made clustering them together fairly straightforward. American Commitment’s clusters were very well behaved because their shared boilerplate was distinctive enough to exclude them from other groupings, hence the large blue supercluster that houses nearly all of their clusters. American Commitment’s tendency to have clusters of approximately 32,000 comments made spotting them easy, too.
Graphic credit: Sunlight Foundation. To embed this on your site, click “embed this widget.” Note: this visualization shows groupings of textually similar comments. Organized letter-writing campaigns that didn’t involve form letters won’t appear here.

For comparison purposes, here are simplified versions of the form letter visuals from parts one and two, side-by-side:

A new get-out-the-comments player

The clear takeaway in examining the comments from round two is the way in which the campaigns we attribute to American Commitment completely changed the balance of opinions expressed. With their comments excluded, the corpus would have looked quite a bit like the first round:

About 728,000 total comments (vs. about 800,000 in round one)

75% of comments would have been form letters (compared to about 60% from the first round)

About 4% of comments would have opposed net neutrality, only a slight increase from the first comment round

Perhaps just as striking as the scale of American Commitment’s efforts was the breadth; most form letter organizers drove large-scale submission of a single comment template, and while many allowed submitters to customize their comments, most submitters apparently chose not to do so. This resulted in one group of nearly identical submissions for most campaign organizers (this kind of behavior is also typical of our experience with form letters in other regulatory arenas). A few more sophisticated campaigners had more than one template, or allowed submitters to plug variant sentences into a single template, but this was generally the extent of the per-submitter variation.

American Commitment, by contrast, had at least 30 different comment variants, many offering wildly different rationales justifying their positions, and taking positions across the political spectrum in their specifics. The number and timing are almost identical across comment templates, which we believe most likely suggests random assignment of prospective submitters to different comment pools, perhaps as a means of testing which messaging drew more submitters, or possibly to try and evade the kind of automated form letter grouping we and others did in the first round. Here is the comment template:

Dear Mr. Wheeler,

As an American citizen, I wanted to voice my opposition to the FCC’s crippling new regulations that would put federal bureaucrats in charge of internet freedom, and urge you to stop these regulations before they’re enacted.

If the federal government goes through these plans to regulate the internet, I know that the internet will change — and not for the better.

[ INSERT VARIANT PARAGRAPH COMMENT HERE ]

Like many Americans, I believe that the internet should remain free of government control and unnecessary regulation — just as it has for the last twenty years of unprecedented growth.

Please stop the FCC’s dangerous new regulations, and protect the future of internet freedom here in America.

Sincerely,

[APPLICANT NAME]

[APPLICANT HOME ADDRESS]

…and here’s a sampling of the variant comments, along with their submission counts and timelines:
The Internet is not broken, and does not need to be fixed. Left-wing extremists have been crying wolf for the past decade about the harm to the Internet if the Federal government didn’t regulate it. Not only were they wrong, but the Internet has exploded with innovation. Do not regulate the Internet. The best way to keep it open and free is what has kept it open and free all along — no government intervention. 150654
Americans have been getting faster and faster Internet speeds because of competition in the free economy, not because of anything the government has done. The Internet does not need the federal government’s “help” and neither the American people nor their elected representatives are asking for the federal government to place political controls over the Internet. The people calling for government control over the Internet are a tiny minority of far-left political activists, and the FCC knows it. Any effort by the FCC to regulate the Internet will be seen by the vast majority of the American people for what it is — another lawless Obama Administration power grab. 32281
The Internet is the biggest economic, intellectual, and artistic success story of the century, and it rose up because of free people, not stifling government. The federal government needs to keep its hands off the Internet. It is not broken, and it does not need to be fixed. It is the federal government, not the Internet, that is broken, and in need of fixing. 32257
Before our government can handcuff a citizen, it must have some reasonable evidence that they have done something wrong. Before the FCC places regulatory handcuffs on Internet providers, shouldn’t the government present evidence that they have actually done something wrong? If the police were to handcuff someone because they might, theoretically, maybe, kind of do something wrong someday, there would be justifiable outrage. Such is the case with the FCC’s attempt to place regulatory handcuffs on Internet providers — just in case they might do something wrong someday. The FCC’s rulemaking in the absence of any actual problem, any actual misbehavior on the part of Internet providers, or any consumer harm is beneath the dignity of an expert agency. 32412
The ideological leader of the angry liberals calling for you to reduce the Internet to a public utility is Robert McChesney, the avowed Marxist founder of the socialist group Free Press. In an interview with SocialistProject.ca, McChesney said: “What we want to have in the U.S. and in every society is an Internet that is not private property, but a public utility…At the moment, the battle over network neutrality is not to completely eliminate the telephone and cable companies. We are not at that point yet. But the ultimate goal is to get rid of the media capitalists in the phone and cable companies and to divest them from control.” In a country of over 300 million people, even an extremist like McChesney can find, perhaps, millions of followers. But you should know better than to listen to them. 32198
Estimating sentiment percentages in non-form comments

Our overall estimate for the roughly 60/40 split between anti-NN and pro-NN was relatively easy to make, since we could confidently classify 88% of comments after reading the 50 form letters that served as their respective prototypes. Still, we were curious to see what the makeup of the non-form-letter comments was. Not only do the remaining documents represent a significant chunk of the corpus, but they’re also potentially the most interesting. These comments reflect the personal interpretations of their authors and give a sense as to how different advocacy messages are shaping how the public thinks about this complex issue.

A brief aside: of the 12% of documents that were not form letters, 14,999 (about 1% of the corpus) looked like this:

To Chairman Tom Wheeler and the FCC Commissioners,

No Content Found — Please specify some content

ruby Michel

33169

This submission is obviously an error. Submissions like this appear as the lone gray circle in the form letter visual above. It appears that all of these submissions were just filled in with name and address information, and no actual content. We were able to locate what we think was the source of this phenomenon: Daily Kos specifically directed participants to write their own comment, rather than using a form letter, in this campaign. It appears that about 15,000 respondents didn’t read the instructions and submitted what were essentially blank documents.

The final 11% of comments (184,120 documents) presented a problem. There were only two of us working on this project, and reading the whole bunch would have kept us busy for quite a while. We decided, instead, to manually read and classify a random sample of 1,840 documents (about 1% of the 11%) to make a training set for an automatic classifier, which is a typical text-mining approach to addressing this type of problem. We trained a similar text classifier in our earlier post to try to estimate the number of expert and non-expert comments in non-form letters.

We selected a random 20% of comments from each of the high-variance clusters, which were predominantly non-form-letter clusters. Of those, we selected a random 1,844 documents to classify by hand. Unfortunately, anti-NN examples were very rare (9 documents) and the rest of the set was split between pro-NN (1575 documents) and those that were either too vague or inscrutable (260 documents). This is data that is too unbalanced for training an automatic clustering algorithm, and so we treated it as a rough estimate of the makeup of the non-form comment pool: 85.4% pro-NN, 14% unclear, and 0.6% anti-NN.

This is hardly a scientific approach, but it’s not very surprising to find a preponderance of pro-NN sentiment in the non-form-letter comments. Free Press organized the submission of over 100,000 comments that included the applicant’s name and a short, unprompted message. Furthermore, as mentioned above, there is evidence that Daily Kos charged its participants specifically to write non-form-letter comments.
Public dialogue or public rant?

Our experience analyzing these comments has given us a unique vantage point on the public’s relationship to regulatory bodies like the FCC, and the role that advocacy organizations play in mediating that relationship. The FCC’s Electronic Comments Filing System is not primarily designed to serve as a platform for debating regulators’ role in serving the public. Nonetheless, when the public was invited to comment upon rules that many believe would have serious consequences for the business community and consumers alike, it naturally gave rise to one of those elusive “national conversations” about a complex and contentious issue.

The term “conversation” might be a bit generous in this case, but if there was one, it’s easy to imagine that the original participant — the FCC itself — might consider it to have completely de-railed. Very few of the comments address specific elements of Chairman Wheeler’s proposed rules. Instead, they focus on the general notion that network neutrality is something that should be either protected or eschewed, depending on a commenter’s personal or professional concerns. These concerns, however, are not always directly relevant to the issue at hand.

On the pro-NN side, arguments include network neutrality’s role in protecting our right to free speech and preventing Internet providers from charging consumers higher fees for faster service.

There can be no freedom if you favor one product over another. Net neutrality is important for protecting free speech, innovation, and healthy competition. Don’t let something this unjust happen to the world. (6018210841-8285)

Without net neutrality, people of the lower middle class wouldn’t be able to afford internet fees, so they’d be stunted on their growth as a race of technology. In this day and age, internet connection is so unbelievably vital to being with the goings on, whether it be email, internet news, articles, job applications, the internet play an enormous role in today’s society, that if most of America couldn’t afford, we’d be setting back our progress as a nation. Plus, you might get riots. (6018211177-9593)

Needless to say, private companies are under no obligation to uphold the First Amendment, and it’s already an ISP’s prerogative to charge its customers more or less according to the speed of their connections. These areas of discussion are at best secondary to the main issues that Wheeler’s proposed rules would tackle. The FCC has also shown no willingness (or, frankly, technological capacity) to fulfill the surveillance-culture nightmares mentioned in other pro-NN comments:

Protect A Free Net. We Don’t Need The FCC To Turn Into An NSA 2.0. (6018211039-5702)

Arguments from anti-NN commenters are at times similarly outside the scope of the FCC’s request. Some commenters seem to understand an “open Internet” to be an Internet without any security:

Open Internet sounds in theory like the right thing to do. Of course! But what about terrorists who creep into our every day lives, no matter how much we protect ourselves? Who’s going to protect the Open Internet?

Anti-NN comments are also sometimes fearful of invasions of privacy:

The internet is fine the way it is. please leave it alone so the common people can enjoy it. big brother NSA should concentrate on the true enemy of the country not all Americans! (6018305588)

But on the other hand, the majority of anti-NN comments seem primarily to take issue with the fact that the FCC regulates anything at all:

I do not understand my government’s “need” to fix what isn’t broken. Please keep your hands and your laws off the Internet. I see the Internet as a place where the best and worst can exist side by side without hurting anyone. Please, considering it is possibly the last vestige of free speech in the world, allow it to create itself according to the needs of its varied users. Thank you. (02-047-005216)

As is often the case with complex issues in the public sphere, framing is everything. Both sides in this digital debate appeal to universally cherished values like freedom, personal choice, security, and economic prosperity. It’s easy to see how those foundational American ideals can be used to generate the submission of millions of passionate responses. What’s less clear is whether or not these concerns, often tangential to the issue at hand, are likely to aid the FCC (which, as we’ve pointed out before, is under no obligation to read all comments) in making its final ruling.
A note about data quality

As with the first round of filings, the number of comments we’re including here is short of what the FCC says it released. The bulk download on which this archive was based contains, according to the FCC, about 2.5 million comments, but as best as we can determine, there simply aren’t that many comments in the archive. It’s difficult for us to be sure, however, because the format in which the comments were released was extremely challenging to parse.

The first seven files in the zip archive contained about 725,000 comments, which aligns with what the FCC announcement told us was the number of submissions posted to the agency’s Electronic Comment Filing System. But the FCC also said it is including email comments that didn’t make it into their main system, and we surmise that the remaining files in the zip archive were these comments. This chunk of comments, however, was concatenated together and then arbitrarily chunked into output files, with no delimiter characters between either one comment and the next or one metadata field and the next, such that it was almost impossible to separate comments from one another.

As was true in round one, we fail to see how the FCC arrived at the count that was widely publicized. Clearly, 1.67 million documents is far short of 2.5 million (the number reported in the commission’s blog post). We spent enough time with these files that we’re reasonably sure that the FCC’s comment counts are incorrect and that our analysis is reasonably representative of what’s there, but the fact that it’s impossible for us to know for sure is problematic, and while we laud the FCC for its good intentions in releasing this data in bulk, we expect better-quality releases from federal agencies to the public. The technical difficulties plaguing the FCC that have hampered their collection of public feedback in this rulemaking are, at this point, well-documented, and it’s clearer now than ever that the FCC needs to make a serious investment in technical infrastructure if it wants the community to seriously engage with its data. Thankfully, it seems that FCC technical staff is aware of these problems, because this kind of release just isn’t good enough.
Comment data

As with the first round, we’re pleased to make available a cleaned up version of the bulk comments for this round of comments. We’ve split the comments from the FCC dump into individual JSON files (one per comment), including both the ECFS comments and the mangled email messages, and also parsed and split an aggregate submission from FreePress representing several thousand comments that showed up as one unintelligible comment in the FCC data.

As a general rule in processing and counting documents, we treated each document submitted to ECFS or received by email as one submission. In certain cases where it was clear that a single submission contained large numbers of distinct comments aggregated together, we made a best effort where feasible to separate those comments into individual records in our data. Petitions, or other circumstances where a single comment was paired with a list of names, were treated as single comments.

We weren’t able to explore many of the ideas we had during the first round about possible avenues for further investigation, and would heartily encourage researchers interested in this data to download the scrubbed versions and consider doing so.
Acknowledgements

We’d again like to thank Radim Řehůřek, maintainer of the gensim library, which was crucial to our text analysis.
Tags:

5 Amazingly powerful Python libraries for data science

5 powerful python librariesIf you have decided to learn Python as your programming language.

“What are the different Python libraries available to perform data analysis?”

This will be the next question in your mind. There are many libraries available to perform data analysis in Python. Don’t worry; you don’t have to learn all of those libraries. You have to know only five Python libraries to do most of the data analysis tasks. I will give a short introduction to each of these libraries, and I will point you to some of the best tutorials to learn them.

So let’s get started,

Numpy

It is the foundation on which all higher level tools for scientific Python are built. Here are some of the functionalities it provides:

N- Dimensional array, a fast and memory efficient multidimensional array providing vectorized arithmetic operations.
You can apply standard mathematical operations on arrays of entire data without writing loops.
It is very easy to transfer data to external libraries written in a low-level language (such as C or C++), and also for external libraries to return data to Python as Numpy arrays.Linear algebra, Fourier transforms and random number generation

NumPy does not provide high-level data analysis functionality, having an understanding of NumPy arrays and array-oriented computing will help you use tools like Pandas much more effectively.

Tutorials

Scipy.org provides a brief description to Numpy package.
Here is an amazing tutorial that completely focuses on usability of Numpy

Scipy

The SciPy library depends on NumPy, which provides convenient and fast N-dimensional array manipulation. The SciPy library is built to work with NumPy arrays, and provides many user-friendly and efficient numerical routines , such as routines for numerical integration and optimization. SciPy has modules for optimization, linear algebra, integration and other common tasks in data science.

Tutorial

I couldn’t find any good tutorial other than Scipy.org. This is the best tutorial for learning Scipy.

Pandas

It contains high-level data structures and tools designed to make data analysis fast and easy. Pandas are built on top of NumPy, and makes it easy to use in NumPy-centric applications.

Data structures with labeled axes, supporting automatic or explicit data alignment. This prevents common errors resulting from misaligned data and working with differently-indexed data coming from different sources.
Using Pandas it is easier to handle missing data.
Merge other relational operations found in popular databases (SQLbased, for example)

Pandas is the best tool for doing data munging.

Tutorials

Quick intro to pandas
Alfred Essa has a series of videos on Pandas. These videos should give you a good idea of basic concepts.
Also don’t miss this tutorial by Shane Neeley, this video gives you a comprehensive intro to Numpy, Scipy and Matplotlib.

Matplotlib

Matlplotlib is a Python module for visualization. Matplotlib allows you to easily make line graphs, pie chart, histogram and other professional grade figures. Using Matplotlib you can customize every aspect of a figure. When used within IPython, Matplotlib has interactive features like zooming and panning. It supports different GUI back ends on all operating systems, and can also export graphics to common vector and graphics formats: PDF, SVG, JPG, PNG, BMP, GIF, etc.

Tutorials

Show me do has a good tutorial on Matplotlib
I also recommend the cook book from pack publishers. This is an amazing book for someone getting started in Matplotlib.

Scikit-learn

Scikit-learn is a Python module for Machine learning built on top of Scipy. It provides a set of common Machine learning algorithms to users through a consistent interface. Scikit-learn helps to quickly implement popular algorithms on your dataset. Have a look at the list of algorithims available in scikit-learn, and you can quickly realize that it includes tools for many standard machine-learning tasks (such as clustering, classification, regression, etc).

Tutorials

Introduction to Scikit-learn
Tutorials from Scikit-learn.org

Conclusion

There are also other libraries such as Nltk(Natural language Tool kit), Scrappy for web scraping, Pattern for web mining, Theano for deep learning. But if you are getting started in python, I would recommend you to first get familiar with these 5 libraries. I have mentioned the tutorials that are beginner friendly, before going through these tutorials ensure that you are familiar with basics of python programming.
Share this:

LinkedIn24
Twitter10

Jetty

Jetty provides a Web server and javax.servlet container, plus support for SPDY, WebSocket, OSGi, JMX, JNDI, JAAS and many other integrations. These components are open source and available for commercial use and distribution. Jetty

Jetty is used in a wide variety of projects and products, both in development and production. Jetty can be easily embedded in devices, tools, frameworks, application servers, and clusters. See the Jetty Powered page for more uses of Jetty.

The current recommended version for use is Jetty 9 which can be obtained here: Jetty Downloads. Also available are the latest maintenance releases of Jetty 8 and Jetty 7.

The Jetty project is hosted entirely at the Eclipse Foundation and has been for a number of years. Prior releases of Jetty have existed in part or completely under the Jetty project at the Codehaus. See the About page for more information about the history of Jetty.

Features Jetty Powered
  • Full-featured and standards-based
  • Open source and commercially usable
  • Flexible and extensible
  • Small footprint
  • Embeddable
  • Asynchronous
  • Enterprise scalable
  • Dual licensed under Apache and Eclipse

You can benefit from committer knowledge and get training, consulting services, professional support and even production SLAs, just ask us about it!