Morning all!

I'm going to be taking the site down for about 5 minutes, so that I can get a consistent copy of our databases (postgres + pict-rs sled).

Will do it at about 10am PST.

Hope nobody missed anything during the outage

Hey everyone, and happy new year!

Sorry about that super long downtime there. Yesterday (Sunday) morning at 10:03AM PST our server suffered a physical hardware failure, apparently a power supply failure. Unfortunately despite opening a ticket with our hosting vendor (OVH) a few minutes later and them claiming to have 24/7 support, nobody looked at our ticket until this morning when their phone support lines opened and I called them.

They've now replaced a defective power supply and we're back online, after ~26 hours of being offline. Some pretty disappointing response times, to put it nicely.

We're planning to move away from OVH at the end of this month, onto proper enterprise grade hardware that we own and control. This will give us a HUGE boost in server resources and allow us to scale for the foreseeable future, while also giving us the control to resolve problems like this much quicker. Expect another follow up post about this in the next couple weeks once I've put together the migration plan.

Timeline:

• Jan 5th 10:03am PST - We get alerts to the server being non-responsive.
• Jan 5th 10:05am PST - I pull up the console via IPMI and it's completely non-responsive. Attempting to power off / on the server or do anything, does not work.
• Jan 5th 10:15am PST - Initial support ticket created with OVH. I followed up a couple times over the next few hours, and got no response.
• Jan 6th 6:32am PST - Called OVH, gave them the case number and asked them to investigate
• Jan 6th 7:34am PST - I get notified they'll start their "intervention" in 15 minutes.
• Jan 6th 11:04am PST - Call them again, the tech is still working on it and they'll get back to me with an update
• Jan 6th 11:34am PST - "I was informed by our data centre technician that there is an issue with the power supply unit for the rack on which your server resides. Your server will come back online once they have replaced the power supply."
• Jan 6th 12:17pm PST - We're back up finally!

Edit on Jan 7th @ 8:40am PST: We just had another outage of about an hour. Investigating with OVH.

Hi everyone!

Last year lemmy.ca ran a census for our instance, and this year we wanted to solicit feedback ahead of time. Since other instances expressed interest in running similar censuses, having some agreed upon questions would make it easier to do comparisons. (previous: lemmy.ca results, lemmy.nz results).

Timeline wise, for our instance, we are thinking of gathering feedback for the rest of the month and then running the census early in the new year. We are open to adjusting when we run it, to align with other instances.

General areas of feedback:

Self Identification / Categorization

With some questions, it is difficult to have a predetermined list of options. Last year we included a free form entry option, and then created word clouds with the full list. However, that is not as insightful, and it takes a lot of time to process. For this year, do you prefer:

• A set of options + a free form option (same as last year)
• A set of options only

Please share if you know of a resource that has a good/inclusive set of options for the question types. Even if a free form option is included, a better set of options as a base will lead to better data.

For context, these were the questions where this happened:

• What is your ethnicity?
• How do you identify? (sexual orientation)
• What is your occupation / field of Work?

Data collection & Results

Last year we used Google forms for collection, and a spreadsheet for calculations and analysis. Visualization was done with google sheets, an open source word cloud generator (github link), and DataWrapper (link) for an interactive display on our website.

This year the plan is similar, although we’re interested to try chart.js for the visualizations. We had some issues with DataWrapper displays, and having the code be open source would make it easier for others to replicate. One consideration is data privacy (do we commit anonymized data to the repo, host it elsewhere and import it, etc.).

Sometime in the future, it might be nice to have a template and/or pipeline that everyone can use, for consistency and to minimize work. In the meantime, do you have any thoughts or recommendations?

Fun Question

Last year we asked about people's favourite dessert. It would be cool to do something similar this year. Do you have any suggestions for a fun question? Some ideas:

• Favourite comfort food
• Favourite fruit
• Favourite color (with a hex code input)

Proposed Questions

Where applicable, I’ve included changes from last year. Depending on the type of instance you run (ex. regional or not), you may want to skip, add, or modify questions.

Section 1: Where is everyone from?

Where are you from?

• Canada? (yes / no)
  • If in Canada, which province or territory?
  • If not, which continent?

What is the size of your community? (rural / urban / suburban / other)

• suburban and other are new options

In your opinion, what is the quality of your internet connection? (poor, reasonable, excellent)

What is your internet speed?

• new question inspired by lemmy.nz
• we may include a link to a speed test site for convenience, and have a free form entry

Section 2: Who are you?

What age range do you fall into? (0-9, 10-19, etc.)

Ethnicity? (options uncertain, see above)

Gender Identity? (Man, Woman, Non-binary person, Genderfluid)

• Optionally, we could include a free form entry 'other' option, and list the input in a separate list.

Are you someone with trans experience (yes / no)

How do you identify? (options uncertain, see above)

Disability Status (yes / no)

• New separate question inspired by lemmy.nz

Disability Status (free form entry)

• Last year we manually processed the text inputs and displayed them in a list. We're open to hearing suggestions on better ways to run this question!

Educational Background

• Last year we had multi-select, which didn't make sense. This year we can split it into a few questions:
  • Are you currently in school? (yes / no)
    • What are you studying? (single selection, see list below)
  • Highest level of education completed (single selection, see list below)
  • All past education (multi-select, see list below)
  • List of options:
    • No formal education
    • Some secondary school (high school)
    • Secondary school (high school) diploma
    • GED
    • Trade / technical / vocational training
    • Some Post-secondary (college, university, CÉGEP)
    • College diploma
    • Associate degree
    • Bachelor's degree
    • Master's degree
    • Professional degree (MD, JD, MBA, etc.)
    • Doctorate (PhD, etc.)
    • Other

Employment Status (full-time, part-time, self-employed, student, retired, unemployed, other)

• Last year we also got responses for 'homemaker', 'disabled', 'active duty military'. We could add these as options, or include a free form 'other' entry for them.

Field of Work (options uncertain, see above)

• This question was a free form entry last year, and it was very difficult to process. Having a solid list of options would be ideal in this case.
• We could include an 'other' option with a free form entry, with a note that any response will not be included in the main results and will be displayed as-is in a separate list.

Section 3: Instance Usage

How many {instance name} communities do you participate in? (0, 1-9, 10-19, 20+)

If you moderate any communities, how many? (number input)

• Last year this was a multi-select, but a number input would be more straightforward

What community do you want to see on {instance name}? (free form entry)

What existing community do you want to see more people using? (free form entry)

• This is a new question.

On average, how much time do you spend on Lemmy each day? (<1 hour, 1-2 hours, 2-3, 3-5, 5-8, 8-13, 13-21, 21+)

How often do you visit {instance name}? (multiple times a day, daily, weekly, monthly, rarely)

• This is a new question.

How do you access Lemmy most often? (desktop, mobile (including tablet), both about equally)

• (including tablet) is a new addition
• both about equally is a new option

If you use desktop, what operating system do you use? (Windows, MacOS, Linux, I don't use desktop, other)

• New question

On desktop, which interfaces do you use? (default, Mlymym, Photon, Voyager, Alexandrite, I don't use desktop)

• I don't use desktop is a new option, and will be separate from the 'skip this question' option

Is there another interface you would like us to add? (free form entry)

• New question

If you use mobile, what operating system do you use? (Android, iOS, I don't use mobile, other)

• I don't use mobile and other are new options

On mobile, how do you access Lemmy?

• We will include an updated list of apps, and a free form entry for any new ones
• We will include an option for I don't use mobile, web browser, and other, in addition to the skip this question option

If you primarily access lemmy.ca through different Lemmy/Kbin instance, please list it below (optional) (free form entry)

What other decentralized / federated platforms do you use?

• Some potential options: none, Mbin, Kbin, PieFed, Mastodon, Bluesky, Pixelfed, Peertube, Loop, Bookwyrm, WriteFreely, Pleroma, Friendica, Misskey, Writefreely, + free form entry

Fun Question (yet to be determined)

Feedback (free form entry)

General Changes:

• All questions will have a 'skip this question' option
• Fewer random comments from me when reporting the results. I looked back at them afterwards and thought they might be a bit much.

One of the drives in our server has failed. =( Even though it should be a 10 minute job OVH needs a 2 hour window to replace it.

I've requested they schedule it for Tuesday from 8am - 10AM PST. Hopefully it'll be reasonably quick, but expect cloudflare tunnel errors while they perform the work.

Hey All!

I'm going to upgrade us to 0.19.7 tomorrow (Sunday Nov 24th) around 10am PST. I don't expect significant downtime, but expect a few minutes at least.

Update: All done!

Hi, we've noticed some server instabilities around 9:20AM (Eastern Time) this morning and it seems our hosting provider had some issues, which lasted for about 40 minutes.

We also took the time to reboot the server after the event just to make sure we're not running into other issues down the road, sorry for the downtime everyone.

Recently I was asked if we'd be willing to reconsider federation with Hexbear, followed by CARCOSA reaching out to me to see if it was something we'd be open to.

They made a post asking if their community was interested in refederating with us, similarly I'd like the opinion of our local users.

Why refederate?

Hexbear has a decent number of sizable active communities:

• https://hexbear.net/c/games
• https://hexbear.net/c/technology
• https://hexbear.net/c/movies
• https://hexbear.net/c/food
• https://hexbear.net/c/vegan

These all seem fairly well moderated, giving us access to more diverse posts and viewpoints. A lot of our feed right now is dominated by US centric communities, and it'd be nice to get some more variety.

Why not?

They love trolling and I don't expect them to follow our rules when commenting (primarily "be civil").

Just looking at their thread about us, there's a significant number of immature replies slandering us / Canada. I'm not confident their users will engage in actual discussion in our threads, rather than just showing up to throw out insults and memes.

We can try to handle this by banning users / blocking specific communities, but that's going to increase moderation workload for both admins and mods. Hexbear is self admittedly quick to ban users on their side, but they also don't mind ban evasion - not ideal for us.

So what's next?

This is where we solicit your opinions, how does our community feel?

Edit: Well, it looks like the community is overwhelmingly against refederation. Decision made, thanks everyone for your feedback!

Currently the instance is the source of unmoderated trolls, and the instance admin seems unwilling to take action despite multiple reports sent.

We're therefore taking the action of defederating. We'd rather work with the instance admin to keep those trolls in check and minimize the disruption before taking a drastic action like this.

For transparency purpose, you can see which instances we are defederating from by accessing the https://lemmy.ca/instances page and going to the Blocked instances section.

We recently crossed the first anniversary for many of us on Lemmy and the Fediverse, and we are thrilled to announce an important milestone: we have started a new registered not-for-profit organization! The organization is registered as ‘Federated Open Communications Canada’, and we will go by 'Fedecan' in our day-to-day operations.

Over the past few months, we have been working behind the scenes on the necessary paperwork, bylaws, and other administrative tasks to get the organization rolling. Thanks to the invaluable feedback from our previous check-in, we decided to take the leap and register federally, which will allow us to operate across Canada more easily. For more details, you can check out the FAQs on our new website.

Resuming Donations

With this important step completed, we are now ready to resume accepting donations. We have set up several donation options to accommodate different preferences on convenience and fees. Our first priority is to reimburse those who have been covering server and other costs, and to set up regular financial updates. We are open to adding more donation options, so please let us know if there is an option you prefer.

• Donation Page: (fedecan.ca/donate)

• Alternate Donation page without embedded elements: (fedecan.ca/donate-private)

Our current costs

• Server hosting - $150/month
• Domains registrations - $50/year
• Non profit requirements - $500 one time

Our Next Steps:

• Reimburse individuals who have covered server and operational costs and to set up more regular financial updates
• Collaboratively creating guides and learning resources about Lemmy and the Fediverse (ex. how things work, data/privacy details, and best practices for moderating or running instances, etc.) + translations
• Exploring other platforms to run. Some that are on our radar include Sublinks, Pixelfed, and Matrix
• Developing logos, colours, and other branding details based on ideas from the community

What we want to know:

• Do you have any questions, comments, or concerns? This will also help us expand our FAQs
• Which other platforms would you like us to explore?
• Do you prefer any other donation method?

We want to help build a better online experience for everyone, and that requires an organization with a strong foundation. This should be the first step in that process.

Thank you for everything so far, and we look forward to seeing you around! 🥳

The Fedecan Team

Thank you to everyone that participated, it was fun :)

Link to the location:

https://canvas.fediverse.events/#x=-54&y=-185&zoom=11

LINK: https://canvas.fediverse.events/

Feel free to use this thread to discuss ideas and otherwise coordinate.

What is it?

You can sign in with almost any Fediverse account (ex. your lemmy account, mastodon account, etc.). Afterwards, you will be able to place a pixel every 30 seconds or so. By working together, people make (and break) different designs.

You shouldn't need to give away your login information to authenticate. For me, the system asked for my instance and username, then sent me a message with the verification code.

Timeline: 72 hours

• Opens on July 12th, 2024 @ 4am UTC
• Closes on July 15th 2024 @ 4am UTC

What should we do?

I don't think we have any templates yet, so you're open to suggest a Canada wide design, or smaller specific ones. In the meantime, just have fun :)

You can see what other people are up to here: https://programming.dev/post/16720673

You can see last year's design here: https://toast.ooo/post/288913

You can use this post as well, but it might be helpful to have everything in one place:

https://lemmy.ca/post/24637498

Hi everyone,

We'll be undergoing maintenance on Saturday morning from about 11am - noon PST.

We've been having some stability issues over the past few days that I've tracked down to database performance, so I'll be doing some tuning and also applying all our needed system updates at the same time.

Cheers!

Update: This is all done now!

Happy Canada Day everyone.

Related to the outage that happened last night, we rebooted the Lemmy services but we're still trying to figure out the root cause, which seems to point to an out of memory issue in the logs. However it's not what we see in our monitoring console.

In the meantime, we will monitor the service more closely until we are confident the issue is resolved, and we will improve our tools to detect such a problem faster.

EDIT: Also happened at night on July 2nd, still trying to find the root cause..

Apologies for the extended downtime.

It may be a few hours till things are up, please bear with us in the meantime.

See status here: https://status.lemmy.ca/

It looks like the alternative frontends are (partially) working:

• https://old.lemmy.ca/
• https://photon.lemmy.ca/
• https://alex.lemmy.ca

Also happy Canada Day!

Back when I joined lemmy.ca, the front page was full of bicycles, city/province-specific community posts, and others like woodworking This morning, four or five of the front page posts were making fun of some crazy bikini lady(?). I read their “read this first” post and understand why they’re doing what they’re doing, but is there a way to get our default front page to be a bit more…friendly and community-oriented?

Sorry for the brief bits of downtime here and there, I snuck in a quick upgrade of us to the latest lemmy 0.19.4 now that a few bugs have been patched out.

More details here! https://join-lemmy.org/news/2024-06-07_-_Lemmy_Release_v0.19.4_-_Image_Proxying_and_Federation_improvements

The sole moderator doesn't even follow their own rules: https://lemmy.ca/post/22741340?scrollToComments=true

I'll just say it - it's a Russian propaganda community. Is there any reason this community needs to exist on Lemmy.ca? Is there a rule against blatant astroturfing / propaganda / misinformation? I don't think the 5 rules in the sidebar are going to be enough to stop an army of trolls:

No bigotry - including racism, sexism, ableism, homophobia, transphobia, > or xenophobia. Be respectful. Everyone should feel welcome here. No porn. Use the NSFW tag when needed. No Ads / Spamming. Bot accounts need to be flagged as such in their settings.

Maybe time to get ahead of it?

I'm in the process of upgrading us to the latest 0.19.4 release.

https://join-lemmy.org/news/2024-06-07_-_Lemmy_Release_v0.19.4_-_Image_Proxying_and_Federation_improvements

You might see some brief downtime during the next hour or so, pictrs is currently upgrading its database.

Edit: Holding off on lemmy as Blaze was nice enough to flag some new bugs - https://lemmy.ca/comment/9660430

Edit: Pictrs upgrading taking longer than expected after the migration crashed, hopefully wrapped up soon!

Finally Done! We're all ready for the new lemmy version now, maybe once a patch release comes out.

I have just noticed a new community that is dedicated to harrassing an influencer they don't like. They are migrating here after being banned from Reddit.

I don't see anything in the rules that this might run afoul of, but I wanted to make a post and ask if this is content we really want on lemmy.ca? The same influencer who got them kicked from reddit could come after lemmy.ca.

The community is !thebirdspapaya_snark@lemmy.ca

Hello,

I noticed that I cannot see some posts or comments from this account on this instance. Some of which are my own posts/comments. These are not posts/comments that have been removed nor does it have to do with bans or blocks.

My own posts/comments for which I receive replies, I will see them in the "messages" section and will be able to access them from there. The posts in communities will be displayed, but comments will show as unable to view

If I log in using an account form another instance, like sh.itjust.works for example. I can see all the missing posts and comments normally.

Here's an example of a post that I made in the Montreal community. I doesn't appear when using my lemmy.ca account, but will appear when using my sh.itjust.works account.

Community: !montreal@lemmy.ca

Post: REV Saint-Denis | Une entrave dans une portion névralgique indigne les cyclistes

Is there a problem with this instance?