Forwarded conversation
Subject: The database
------------------------

From: Misled Youth
Date: Sat, Jun 14, 2008 at 1:23 AM
To: icarus web , Icarus Project Distro, Brendan Heidenreich , Chris Lumpkin, Molly

Hi everybody. So, with the database there's a whole bunch of different things going on at once. There's the meeting we just had with the guy from Democracy In Action on Wednesday, there's the phone conversation I had with Jonah, there's some good input from Chris (aka jackrabbit) (who's part of the forums and an experienced Java/J2EE/MYSQL developer), there's stuff I've picked up on from the conference calls that are fleshing out icarus' specific needs, there's a few ideas from my roommate Brendan, who knows a lot about php, mysql and general computer-related stuff, and then there's my own ideas and opinions. I'm gonna try to pull all these different things together... this is really testing the limits of email communication... Jonah had some good cautionary advice that I'll start off with: First, we should pick a method of storing data that's sustainable, it shouldn't be dependent on one or two people who knows how it works, it should be as simple as possible, easy to export, and financially manageable. We should also try to prioritize what the web team focuses on, like improving the front-end of the site, setting up organic groups, solving some of the problems tracked on open plans, etc. He also said that if we used something that integrated with the site now, we should be careful not to do anything that would slow it down or expose sensitive information (like email addresses, phone numbers, amounts people contributed) to the public. Chris had some really useful input on the process we should use to create the database. like listing all the things we want to be able to do (system requirements) and creating scenarios for how the user would interact with the database (user stories). Both Chris and Jonah (and me, and probably everyone else) agree that we should really decide on what icarus needs exactly. The conference calls, which for the past two weeks have been between Mimi, Molly, Michael and me (what a tongue twister!), have brought to light some needs. Contacts, donors, email blasts and purchases of media are basically the number-one urgent things we need to track. However, every working group could really benefit from a comprehensive, easy-to-use CRM. Communication could really be improved, tracking of work-group needs, goals, volunteers, payrolls, etc. Having some sorta degree-of-urgency feature for emails and way of naming shared files something coherent would be helpful (Molly's suggestions). The meeting with Charles from Democracy in Action brought up a whole lot of things. One upside to using their database software is they can collect donations and payments for materials. Right now, FJC collects the payments, which means they collect the information from the person paying. They do a terrible job displaying this information to Icarus, they only really show the name, date and amount people donated, Mimi's been having to email them for the contact info of people. This really sucks for people who are requesting a reader or something, because we can't send it to them without collecting their address. With Democracy in Action, the money could go to them first, they collect all the information from the person, then send a check to FJC, and FJC gives Icarus the money. one problem with this is both Democracy in Action and FJC each take a cut of about 3%, which comes out to be 5.91%, which is pretty high. for every $17 donated icarus would be losing $1. A number of ideas rose out of this, one being Icarus tries to use a non-profit that was set up in Massachusetts a few years ago as it's the umbrella instead of FJC. Another was to set up Icarus as a non-profit itself, though this might conflict with a previous decision that Icarus didn't want to be a 501c(3) because it would restrict some of the stuff icarus could do. Both ideas would amount to a new accountant role. Another idea is to have people submit their information to us, then be transferred to FJC to complete the transaction. I know with paypal you can collect information from someone on your own site, then direct them to paypal along with the info they've already submitted (like name, email, amount, etc.) and they don't have to write all that info out again. I don't know if you can do that with FJC's payment process, but maybe. The only catch is, if someone filled out the form on icarus and then didn't complete the transaction they'd still be in the database as having donated or payed for a book. Something could possibly be set up to check whether or not they're also in FJC's table of completed transactions, or someone could just check off a 'payment received' checkbox on the icarus site once the check comes through. Anyway, so after that there was also some appeal to Democracy in Action in its speed and it seemed to generally handle all of the things Icarus needs. They have some disadvantages though, in that they would not provide any new features or help outside of basic trouble shooting, they will only correspond with one person, salsa is not open source, and the stuff that we already knew about how they cost all this money and would not integrate with the site. Brendan, who's a friend of mine and lives with me (and Michael and Sarah), had an idea that if we set something up ourselves we could submit it as an open source project and gain help from other people in need of the same thing. That would help in terms of sustainability. Ok. I think that covers all the different pieces. First off, I have some idealistic problems with how Salsa's not open source. Democracy in Action says that the reason they do what they do is to put more power in the hands of organizations, but meanwhile they're keeping the core of that power to themselves for the explicit purpose of making money off of it. They could at least have it be open source and charge money for services and hosting, many CRMs seem to do that. Also, they say they built it by embracing open source projects, which means they're taking code that was intended to be used and developed freely and charging people to use it, not allowing people to develop it, and not crediting the authors of the specific code. But ideologies aside, I still think Salsa is an unnecessary expense. Free solutions exist, whether it be using drupal itself or civicrm or google spreadsheets or basebuilder or something else. Particularly because Salsa doesn't integrate with anything without a lot of custom code that they wouldn't help us write or troubleshoot, we might as well use a free solution that doesn't integrate with anything, we could get essentially the same thing. There are free open source CRMs in existence, like cream, splendidCRM, and sugarCRM (which has a free version and a 'professional' not-so-free version). granted, I haven't really looked into any of them, but I bet they'd provide similar features to salsa. If integration is not a great concern, I'd say one of those is our best option. However, I think integration is important. While things like iframes and redirects can provide pseudo-integration, I think real integration with drupal would make the CRM more adaptive in the long run, specifically because of the endless variety of modules that can enhance whatever we set up in the future when new needs present themselves. I think a particularly interesting idea was Brendan's that we could create an open source project out of this. Specifically, I think making a CRM module for drupal that was lighter than civicrm and creating a project on drupal's website for it would be good. I think the drupal community would probably help in terms of development and support, I know we weren't the only ones who thought civicrm was clunky and confusing. And while with salsa their support is limited to a very small team and only one person can ask for help and they'll never add features, if we used the drupal community there would be tons of programmers ready to help outta the goodness of their hearts and there own interest in developing it. [Now I'm gonna get a little technical... just warning you] Regarding Jonah's concern about speed, queries slow down the site. Every search, edit and insertion of data creates at least one query, usually more. Every refresh of a page results in numerous queries, and particularly the more views of data you have on a page the more expensive it is to refresh it. By using ajax wherever possible (which, to those not technically inclined, is a technique that basically does something on the server without reloading the page) many unnecessary queries can be avoided. By removing some of the many blocks currently on every drupal page, or at least confining them to pages that really need them, tons of queries can be avoided on the front end. Inserting lots of data at once can be somewhat slow while you're doing it, but as long as most of that data goes into its own table (like supporters go into the 'supporter' table) it shouldn't slow down the front-end site when someone does something that searches, say, the 'users' table (however a row does go into the 'node' table when using cck that contains information like the author and the date it was created. however, it could just not do that, it could just go into the supporters table and not even make a node, but that again depends on how integrated we want the crm to be). But yeah, the more it's integrated with the site the slower the crm itself will be. It's a trade-off, and it's the reason why salsa can be so fast-- salsa takes a command and does a few direct queries, drupal takes a command and it's filtered through its core system and all the modules which might take some of the information and insert it into there own tables. As far as how portable the data drupal stores is, I can't see a reason why you wouldn't be able to just export it, even if it references other tables. With, as an example, the supporters table referencing the users table, the donation table, the node table, etc. you could just join them by there common node id or user id, right? I guess it's worth looking into whether or not cck stores data as name-value pairs in Drupal 5. Johah says this is the case for drupal 4.7, I say it's not the case in Drupal 6, but what matters is Drupal 5. I can check this out... Ok, just checked it out. Each content-type gets its own table, with a column for each field type, a column for 'vid' (categorization id) and a column for 'nid' (which connects it to the node table). I think this is because starting with Drupal 5 they moved the 'create new content type' cck feature into drupal core, which probably created a better storage system for it in the process. Is this portable enough? Would something else would be any more portable? I think keeping sensitive information about users is fairly simple. Drupal's access control can lock off node types, and entire sections can be hidden from users who don't have a certain role. However, we could make it so the user can view their own supporter/donor information, like as a tab next to their profile. We could also let them edit some of their own information in case they move or change their email address. In terms of priorities, I'll leave that to everyone else to decide. Should the web team focus on the database issue or setting up organic groups? Both would be really useful tools for all the new working groups... I could go either way. I think that just about covers it, and the size of this email's getting ridiculous. We should figure out a better way for all sides to communicate... Chris made a good point that this isn't a linear conversation but email is a somewhat linear format. Would a private forum be better? Thanks everybody for your help with this. There are quite a few knowledgeable voices in this discussion now and I'm sure we'll come up with a good solution. I'm adding Brendan to the list of receivers, could someone add him to the web-group list, or better yet send me the login info to do that myself? -Nick

----------
From: empties
Date: Sat, Jun 14, 2008 at 2:45 AM
To: icarus web
Cc: molly, Icarus Project Distro, Brendan Heidenreich, Chris Lumpkin, nick

Wow Nick, I see you've put a lot of thought into all this. A lot of it is beyond my realm, but just to quickly respond to a couple of things: We can set up a private forum easily enough. Everyone would need a forum account, and then we'd add them to a private usergroup. It would be hidden from the rest of the users. No trouble at all, if that's what people want to do.Hi, Brendan. You should be added now. The info page for that list is here by the way: https://lists.riseup.net/www/info/icarus-web That is all for now... --- On Sat, 6/14/08, Misled Youth  wrote: From: Misled Youth Subject: [icarus-web] The database Date: Saturday, June 14, 2008, 2:53 AM

----------
From: Jonah Bossewitch
Date: Sat, Jun 14, 2008 at 8:07 PM

Hi Everyone -

Communication's characteristic properties are multi-dimensional, and
there is more that we need to balance than simply linear and non-linear.
 Public and Private as well as Push vs. Pull come to mind, offhand.

I want to make an emphatic plea for us to continue this conversation
using the existing sympa mailing list.

The emails sent to the icarus-web address are managed by the Sympa
mailing list software. It threads, archives, web-publishes, and even
exposes rss feeds. In many respects it offers a superset of features as
phpbb - if you prefer to follow the conversation threaded over the web,
then just visit this page:
But, there are many of us who do not check the forums regularly who like
to follow this conversation. Personally, I find phpbb conversations
/very/ difficult to follow, especially for task oriented work. Also,
this list has been open (or, at least openly joinable) up until now, so
I don't know why we would close it.

But seriously, I would say that especially for technical matters, which
are often time sensitive, we need a push communication channel.

I hope this doesn't start any religious wars, but really, this list has
been working well for me (us?) for quite some time - is there really a
compelling reason to change things, other than the vague desire for
"non-linear" communication?

mad love,
/Jonah


----------
From: will hall
Date: Sun, Jun 15, 2008 at 2:11 AM

hi nick, this is a great discussion and thanks for your detailed consideration. also chris thanks for your involvement this is really great to have your participation, and also brendan hi there good to have you involved.

i'm doing a lot less with the web team these days, but thought i would chime in on this dialog. a little history -- the decision to go with drupal for the site was a long discussion in the forums among numerous people, all looking at the overall range costs and benefits. we decided to port the entire site to drupal, which jonah and i worked hard on with edgard durand a volunteer php / drupal guru from mexico city.

one of the main considerations for going with drupal was the idea of extensibility and evolution of function, and the importance of a single integrated platform solution. we felt that this was the most advantageous position for icarus to be in the future. we figured, for example, that by going with drupal we could attract really cool volunteer energy - which has turned out to be really true, with edgard, scott, amanda, and now nick. another main consideration was embracing open source software for political reasons, well as the simple economic reasons of saved money, volunteer coders available, free software add ons etc.

so my sense here is that we might be discussing adding a new room on the house by first discussing where the house should be located, what wood to use, etc. the house is actually already located and the wood already bought. the decision to go with drupal was a very good decision and has served us well; we're only scratching the surface of what drupal can do, and drupal is perfectly positioned in the world of software to grow with the fast changing environment of the web. we also should think about focusing our attention on deepening our overall knowledge of drupal so we have more tech knowledge at our disposal as we develop our web tools,  rather than spreading out over more than one platform.

so my sense is that we should go with nick's enthusiasm for a drupal crm  solution. the watchword of any development he does should be flexibility for expansion and adding features. right now we have very concrete and simple database needs. let's focus on a Stage One plan where Nick works out a simple database that meets those needs and is integrated with the rest of the database. the biggest potential obstacle i see is the multiple queries to the database issue and the lag that could potentially create on adding items and editing the database. that was the problem scott and i ran into with civicrm. but it sounds like this problem can be addressed.

So then once stage one is nailed down, let's have a larger discussion for Stage Two on broader needs and long-range development. Nick you should alert us to any decisions on Stage One that might lock us in to some Stage 2 possibility, and work to keep all our options open.

i also want to caution about anything where we store credit card information. i suggest we use a third party donation management service like http://www.networkforgood.org/ for donations and some simple shopping cart service like paypal for sales. i think we should outsource our distro and sales as much as possible so we don't get into the business of being a storefront, we've got plenty of other priorities and  it makes sense that a distributor that focuses on that will be better set up to do it.

also one thing to consider about the 501c3. Right now icarus uses another organization as an umbrella. i think we should keep doing that. because if we incorporate, we then have to have a board of directors and articles of incorporation etc etc. that would really affect our own internal collective decisionmaking structure. we could of course set it up with two structures, the formal structure to comply with the irs and the actual structure we use. but in practice they would become blurred and confusing. right now fjc has a board etc that we never think about, and we are free to do our experimental collective thing.

just some ideas here and thanks everyone for your involvement...

-- will  >Chris made a good point that this isn't a linear conversation but email >I'm adding Brendan to the list of receivers, could someone --- On Sat, 6/14/08, Misled Youth
----------
From: Chris Lumpkin

I believe in open source for many reasons. The easiest reason is that it's "free like free beer": the software doesn't cost anything. It's also "free like freedom", so if something should go wrong you have the original source code for troubleshooting! It's also code sharing across the entire software industry. You're no longer limited to the talent you can attract to Icarus, or the talent you can afford- you now have the best minds in software, world wide, working for you. Open source is the shizz-nit. If you have some that meets your needs, then ride with the wind at your back.Some degree of written requirements and documentation would prevent us from having this conversation again. Again- it doesn't have to be very structured. We could even cut and paste from these emails, and having a central place to keep artifacts about decisions like this will provide clarity and prevent all these wheels from getting re-invented. Maybe the central place is indeed the riseup list... just so we all know to look back for previously made decisions.


----------
From: will hall
Date: Sun, Jun 15, 2008 at 11:44 AM
To: Jonah Bossewitch

also just to add to the mix of possibilities, 37signals is a pretty established tool that recently announced this online shared contact database manager tool that looks very easy to use:

http://www.highrisehq.com/
----------
From: Icarus Project Distro
Date: Mon, Jun 16, 2008 at 3:13 PM

Hello most thoughtful and talented people,

 

I'd like to weigh in on what I experienced meeting Nick and Charles from DIA last week and where I stand on all this. To give you an idea of where I'm coming from, let me clarify my role in all of this. I was hired in January by the owl collective to coordinate distribution of publications, manage administrative tasks, thank and track donors to the Icarus Project. While I have never built a database and I'm not a programmer, I was hired in part because I have worked a lot with national organizations struggling to keep track of information and using a lot of different solutions to do that. One of the hopes of the collective in hiring me was that I could resuscitate the process of building a database. As Will mentioned this had been an ongoing struggle and a rather frustrating one for some time. The collective hoped that I could use my experience as a grassroots organizer with SOA Watch, War Resisters League, an organization called Groundwork, Bikes Not Bombs and others as well as my connections to tech folks to help build collaboration between what I think you guys call "end users" and those who do the back end stuff.

 

My first step in pursuit of figuring out a solution to Icarus' database needs was to spend my first 6 months in the office assessing the way things work now, talking to organizers and the former office manager Alicia to get a sense of consensus about what our needs are. So I feel I have a really firm grasp on what the hopes and desires of many are with this project, and what it would take to build a really functional national office.

 

Having said all of that, this is where I stand. The biggest thing on my mind is the cautionary tale of War Resisters League. A year ago they set up a database committee that went over some options and decided to hire Mayfirst peoplelink to build a custom database they could query in all these specific ways. These guys are paid professional consultants who work day in day out on nothing but databases and the project they said would take 3 months is still not complete. I have been to the WRL office several times since we've started this and they are adamant that if we have the choice to go with something that is already built, we will be infinitely grateful to avoid the headache. I can't stress enough how important I think it is that we be really realistic about timelines and the time-suck of person power building this could be.

 

Here are what I see as the major advantages of salsa:

1. We need to start fundraising for next year right away. A lot of discussion has happened about being more grassroots in our fundraising, not relying on foundations or large donors or sascha's connections as much. Salsa would be the way to get that ball rolling now, and give more of our base a stake in the work by connecting them to easy ways to donate, volunteer and otherwise plug in. (Right now it is NOT easy to donate, and keeping track of donors through FJC is really labor intensive and ineffective.) I expect this will take a lot longer to get up and running and work out kinks if we were writing the code ourselves. If I understand correctly, in order to even do email blasts with Drupal we'd have to plug in a separate email blasting solution, and there may be security concerns. In addition, emails will likely get blocked by people's spam filters. DIA's emails work with spam blockers to make sure they get into people's inboxes.

 

2. Operating a storefront seems like a really simple necessity. People come to our site looking for a quick way to purchase our publications and currently there isn't one. With salsa not only could we get our materials expediently into the hands of people who want or need them, but we could easily keep track of these folks as potential supporters and consider them part of our movement once they have taken that step. To me spreading the seeds of ideas and connecting with people that way is a core of our mission, more so than developing our own software. I feel that basic function should be prioritized. Even if we were to outsource more widespread distribution, it's important that we can offer some things through our website, even if it's just an "organizer start up kit" so that we can make person to person contact with folks who want to get started doing support groups. We can either use DIA's credit card info management service, or we could sign up with something else like verisign or something more free.

 

3. DIA is a nonprofit and the bulk of the costs are associated not with the software but with support and service. This support enables us to put the tools they have to offer in the hands of the grassroots. If Nick and the web team were to take on this project, would they also be committed to training others (anyone from me in the office to an Icarus group leader in Wyoming for example)? The volunteer management aspect of their jobs practically doubles the workload.

While it's true that DIA prefers to only communicate with one or two people directly, a major goal of their mission is to empower a broad base of folks to use the tools. This means they've spent several years developing the cooperative, people side of the technology. I feel like utilizing their expertise in making the system accessible is a major advantage and also more in keeping with our non-hierarchical structure. Also, they offer everything from forums to web seminars to conferences to their users. So there's a lot more sources for training and troubleshooting than just the consultant we'd work with one on one.

 

4. I think we can achieve an acceptable level of integration with the website and salsa. Anyone who is already registered with the site will be migrated into the database. We don't even have to move it over ourselves, they do it for us. I don't see what more integration we would need that couldn't be solved with the "pseudo-integration" solutions Nick mentioned. We can create pages of our site that look just like the rest of our site that automatically dump info into the database.

 

As far as the cost, I feel strongly that with the boost in income we'd be able to generate with a simpler donation and pubs purchasing, it would pay for itself. They also have a grant program for organizations that can't afford their services that I think we would be an excellent candidate for.

 

As far as the idealogical debate about open source, I don't think it's accurate to imply that DIA appropriates open source code and makes a profit from it. They are a 501c3, they take code that non-tech people wouldn't otherwise know what to do with and translate it into something that is useable by activists out in the streets. Kind of like paying for a translation service, and a support team once you put it in action. Or as they put it:

 

"We're a progressive nonprofit whose core activity is providing e-advocacy tools to other progressive nonprofits for pennies on the dollar relative to the fees demanded by the private sector. We democratize e-activism, freeing practitioners to pour resources into mission and strategy."

 

It's used by all kinds of other radical groups like the Ruckus Society, SOA Watch, Harm Reduction Coalition, Save Access, Brave New Films, and bigger more moderate groups like ACLU and NAACP. They screen their clients and only work with progressive organizations.

 

In terms of us contributing something to the open source movement, I think it would be cool but wonder how it jives with our mission. I know we are an independent media project but I feel like it's not at the top of the list of commitments we have as a collective. Seems like a lot of time and energy to pour into something that's not at our core.

 

All of that said, I'm only trying to facilitate a good solution for the Icarus universe. I wouldn't be opposed to any strategy we agree on including using Drupal, and like I said I have full faith in Nick and all of you and your abilities. I just think this tool is out there, it's a great fit, it's sustainable, meets our needs, it's within our grasp, and would enable us to reach further, go deeper, and focus ourselves on building mutual aid, doing advocacy and shifting the culture of our world gone mad.

 

PS I'm OK with having this dialogue in another forum/format and appreciate your willingness to facilitate that empties, thank you.

Much love to you all for your dedication and awesomeness,

Mimi

----------
From: Chris Lumpkin
Date: Tue, Jun 17, 2008 at 12:16 AM


SOA Watch? Do you know Vera Leone? She's a virtual, soon to be real, friend of mine!

Peace,
Chris
----------
From: Jonah Bossewitch
Date: Tue, Jun 17, 2008 at 1:16 AM


+2

Mimi - this is awesome. By far the clearest and most comprehensive
description of our use cases to date.  In my mind many of these
requirements rule out some of our earlier considerations.

If you have worked with this software before, and are keen on using it,
I am all for beginning negotiations with them to find out if we could
qualify for a grant.

To put this another way, if salsa were gratis, would there be any
remaining doubts? We have the issue of free software, which is
legitimate and has plenty to do with our mission, but on the other hand,
every bit of software we write and run is a liability. Freedom comes at
a price, which we must struggle to balance.

If this database will be running email campaigns, as opposed to just
tracking contacts - to be subsequently fed into another system for doing
the email blasts - we have to think /very/ hard about running that
ourselves. Even though drupal has a module for email blasts, sending
mass emails nowadays is a black art that none of us have expertise with.

While I agree that a storefront is essential, I think a link out to a
place where our products are for sale will be sufficient. We are likely
much better off using a service that will handle on-demand production,
processing, and fulfillment, than taking orders ourselves.

Will - 37 Signals Highrise looks sweet - great find. It doesn't cover
all the bases Mimi lays out here, but perhaps it covers enough of them.
It won't do email blasts though,  so we would need to export names and
feed them to another system to run a campaign.

I am not on the inside of the financing here, but I suspect that if
Icarus doesn't start fund raising soon, the $ will run out - we may need
to take some risks in the hopes of a faster/greater return.

Mimi, how can we find out about their grant program?

mad love,
/Jonah
>    1. We need to start fundraising for next year right away. A lot of
> * *
>
>    2. Operating a storefront seems like a really simple necessity.
>    3. DIA is a nonprofit and the bulk of the costs are associated not
>    4. I think we can achieve an acceptable level of integration with

----------
From: Clare Christina
Date: Tue, Jun 17, 2008 at 7:35 AM


I agree completely.
Because I analogize everything, I'm going to say that our need for these services is like living outside and needing shelter. It's kind of a question of renting a space to live in or building a house ourselves (or having Nick and a few others do the building while we stand around in the rain). Renting a place we can move into immediately means we can focus on the work we do, and as Mimi suggested, it might pay for itself if it means we can draw in more funds through outreach and pub sales. Sure, I'd like to live in a custom built house, but that doesn't seem realistic right now. Especially if the house ever requires maintenance.
One more idea to throw out there: has anyone worked with Tech Soup? They provide massively discounted software to nonprofits. Not that we're a non profit but we might have access to their services.
Clare

 

Forwarded conversation
Subject: databasing proposal
------------------------

From: Icarus Project Distro
Date: Mon, Oct 6, 2008 at 12:01 PM

----------
From: Misled Youth
Date: Wed, Oct 8, 2008 at 12:43 AM


----------
From: Icarus Project Distro
Date: Wed, Oct 8, 2008 at 10:37 AM



----------
From: Icarus Project Distro
Date: Wed, Oct 8, 2008 at 10:49 AM



----------
From: Icarus Project Distro
Date: Tue, Oct 13, 2009 at 10:36 PM


Some history...



----------
From: Icarus Project Distro
Date: Tue, Oct 13, 2009 at 10:36 PM
To: Eric Stiens

Here's the WHOLE thread...sorry about that, Eric!

A